00010000:00010000:1.0F:1781047566.849312:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1781047567.873316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0F:1781047568.897337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1781047569.921323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1781047570.305334:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1781047570.305337:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513309056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047570.305343:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047570.305347:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047570.305354:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1206 00000100:00080000:0.0:1781047570.305358:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00080000:0.0:1781047570.305364:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047570.305374:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047570.305392:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ef3e6f4 x1867562513309184/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047570.305410:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047570.305412:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047570.305428:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513309120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0F:1781047570.305460:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ef3e6f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513309184:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047570.306886:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513309056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0F:1781047570.306971:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047570.306982:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047570.306985:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ef3e6f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513309184:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047570.307016:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513309120:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781047570.307087:0:2216:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 6/2,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307103:0:2216:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 6/2,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307110:0:2216:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 6/2,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307116:0:2216:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 6/2,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307146:0:2216:0:(ldlm_lock.c:909:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 5/1,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307183:0:2216:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 4/1,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307190:0:2216:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 3/0,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307197:0:2216:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 4/0,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00010000:00010000:3.0:1781047570.307204:0:2216:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 0000000055b1cf0b already unused, calling callback (00000000004f6738) 00000020:00010000:1.0:1781047570.307207:0:19116:0:(cl_lock.c:53:__cl_lock_trace()) release lock: 0000000014d75a24 (00000000f720b6d6/0) at cl_lock_release():234 00010000:00010000:3.0:1781047570.307208:0:2216:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 5/0,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00000020:00010000:1.0:1781047570.307213:0:19116:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: 0000000014d75a24 (00000000f720b6d6/0) at cl_lock_cancel():151 00000020:00010000:1.0:1781047570.307220:0:19116:0:(cl_lock.c:53:__cl_lock_trace()) cancel lock: 00000000e65a375d (00000000cd3acd42/1) at cl_lock_cancel():151 00000020:00010000:1.0:1781047570.307224:0:19116:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: 0000000014d75a24 (00000000f720b6d6/0) at cl_lock_fini():84 00000020:00010000:1.0:1781047570.307228:0:19116:0:(cl_lock.c:53:__cl_lock_trace()) destroy lock: 00000000e65a375d (00000000f720b6d6/1) at cl_lock_fini():84 00010000:00010000:3.0:1781047570.307244:0:2216:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00000080:00200000:1.0:1781047570.307245:0:19116:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x33:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0 00010000:00010000:3.0:1781047570.307250:0:2216:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 3/0,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1781047570.307251:0:19116:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x33:0x0] ignore/verify layout 0/1, layout version 2 restore needed 0 00010000:00010000:3.0:1781047570.307262:0:2216:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 0000000055b1cf0b/0x75014a414d98e114 lrc: 0/0,0 mode: --/PR res: [0x59:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19116 timeout: 0 lvb_type: 1 00000080:00200000:1.0:1781047570.307269:0:19116:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x33:0x0] ignore/verify layout 0/1, layout version 2 need write layout 0, restore needed 0 00000100:00100000:3.0:1781047570.307284:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:0:1867562513308992:192.168.201.144@tcp:101:stat.0 00010000:00010000:3.0:1781047570.945238:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1781047571.970343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1781047572.993313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000080:0.0:1781047573.200393:0:19806:0:(module.c:221:libcfs_ioctl()) libcfs ioctl cmd 3221775648 00000001:02000400:0.0:1781047573.200398:0:19806:0:(debug.c:711:libcfs_debug_mark_buffer()) DEBUG MARKER: == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:26:12 (1781047572) 00000080:00200000:0.0:1781047573.219259:0:19809:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:0.0:1781047573.219279:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1781047573.219295:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.219313:0:19809:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1781047573.219325:0:19809:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:0.0:1781047573.219333:0:19809:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:0.0:1781047573.219341:0:19809:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000008b0368f 00010000:00010000:0.0:1781047573.219345:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.219353:0:19809:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1781047573.219369:0:19809:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:0.0:1781047573.219379:0:19809:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), flags=8 00000080:00200000:0.0:1781047573.219384:0:19809:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=getattr 00000002:00010000:0.0:1781047573.219395:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1781047573.219436:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.219444:0:19809:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.219452:0:19809:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1781047573.219989:0:19809:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309248:192.168.201.144@tcp:101:rm.0 00000100:00100000:0.0:1781047573.220274:0:19809:0:(client.c:2500:ptlrpc_set_wait()) set 000000003ff54068 going to sleep for 388 seconds 00000100:00100000:3.0:1781047573.223013:0:19809:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309248:192.168.201.144@tcp:101:rm.0 00010000:00010000:3.0:1781047573.223030:0:19809:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223043:0:19809:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223049:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223055:0:19809:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223062:0:19809:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223067:0:19809:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000126b7fc2 already unused, calling callback (000000002988a2da) 00010000:00010000:3.0:1781047573.223078:0:19809:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223085:0:19809:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223093:0:19809:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223109:0:19809:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1781047573.223115:0:19809:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223121:0:19809:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000126b7fc2/0x75014a414d98e11b lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000002:00100000:3.0:1781047573.223156:0:19809:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@000000005b954604 x1867562513309248/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 584/640 e 0 to 0 dl 1781047963 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:3.0:1781047573.223181:0:19809:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:3.0:1781047573.223231:0:19809:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:3.0:1781047573.223240:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1781047573.223247:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223255:0:19809:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:3.0:1781047573.223264:0:19809:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:3.0:1781047573.223267:0:19809:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:3.0:1781047573.223271:0:19809:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000036c67005 00010000:00010000:3.0:1781047573.223280:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223287:0:19809:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:3.0:1781047573.223299:0:19809:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:3.0:1781047573.223310:0:19809:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), flags=0 00000080:00200000:3.0:1781047573.223314:0:19809:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=getattr 00000002:00010000:3.0:1781047573.223324:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1781047573.223358:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223366:0:19809:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:3.0:1781047573.223373:0:19809:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000100:00100000:3.0:1781047573.223390:0:19809:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309312:192.168.201.144@tcp:101:rm.0 00000100:00100000:3.0:1781047573.223521:0:19809:0:(client.c:2500:ptlrpc_set_wait()) set 000000003e832b99 going to sleep for 388 seconds 00000100:00100000:1.0:1781047573.224878:0:19809:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309312:192.168.201.144@tcp:101:rm.0 00010000:00010000:1.0:1781047573.224891:0:19809:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224899:0:19809:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224904:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224908:0:19809:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224912:0:19809:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224923:0:19809:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 000000004d9c92f6 already unused, calling callback (000000002988a2da) 00010000:00010000:1.0:1781047573.224926:0:19809:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224932:0:19809:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224938:0:19809:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224945:0:19809:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1781047573.224950:0:19809:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.224957:0:19809:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004d9c92f6/0x75014a414d98e122 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1781047573.224971:0:19809:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@000000005840dd0f x1867562513309312/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 584/640 e 0 to 0 dl 1781047963 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:1.0:1781047573.224992:0:19809:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:1.0:1781047573.225037:0:19809:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:1.0:1781047573.225044:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1781047573.225049:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.225055:0:19809:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:1.0:1781047573.225069:0:19809:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:1.0:1781047573.225070:0:19809:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:1.0:1781047573.225073:0:19809:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000008b0368f 00010000:00010000:1.0:1781047573.225075:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.225079:0:19809:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1781047573.225086:0:19809:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:1.0:1781047573.225097:0:19809:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000b5d56edc), flags=8 00000080:00200000:1.0:1781047573.225099:0:19809:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=getattr 00000002:00010000:1.0:1781047573.225105:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1781047573.225178:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.225183:0:19809:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.225189:0:19809:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1781047573.225202:0:19809:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309376:192.168.201.144@tcp:101:rm.0 00000100:00100000:1.0:1781047573.225245:0:19809:0:(client.c:2500:ptlrpc_set_wait()) set 000000000fc4768d going to sleep for 388 seconds 00000100:00100000:1.0:1781047573.226442:0:19809:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309376:192.168.201.144@tcp:101:rm.0 00010000:00010000:1.0:1781047573.226453:0:19809:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226478:0:19809:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226486:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226493:0:19809:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226500:0:19809:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226506:0:19809:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000c7d2d12f already unused, calling callback (000000002988a2da) 00010000:00010000:1.0:1781047573.226509:0:19809:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226515:0:19809:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226523:0:19809:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226530:0:19809:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1781047573.226533:0:19809:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226538:0:19809:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000c7d2d12f/0x75014a414d98e129 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1781047573.226561:0:19809:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@00000000a161b523 x1867562513309376/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 592/640 e 0 to 0 dl 1781047963 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:1.0:1781047573.226579:0:19809:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:1.0:1781047573.226611:0:19809:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:1.0:1781047573.226617:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1781047573.226622:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226628:0:19809:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:1.0:1781047573.226633:0:19809:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:1.0:1781047573.226635:0:19809:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:1.0:1781047573.226638:0:19809:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000036c67005 00010000:00010000:1.0:1781047573.226640:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226644:0:19809:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1781047573.226650:0:19809:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:1.0:1781047573.226657:0:19809:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000b5d56edc), flags=0 00000080:00200000:1.0:1781047573.226660:0:19809:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr-2, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=getattr 00000002:00010000:1.0:1781047573.226664:0:19809:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1781047573.226681:0:19809:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226696:0:19809:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.226704:0:19809:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1781047573.226719:0:19809:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309440:192.168.201.144@tcp:101:rm.0 00000100:00100000:1.0:1781047573.226817:0:19809:0:(client.c:2500:ptlrpc_set_wait()) set 000000003e832b99 going to sleep for 388 seconds 00000100:00100000:1.0:1781047573.227790:0:19809:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job rm:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19809:1867562513309440:192.168.201.144@tcp:101:rm.0 00010000:00010000:1.0:1781047573.227802:0:19809:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227810:0:19809:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227816:0:19809:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227822:0:19809:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227826:0:19809:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227831:0:19809:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 000000004dd5eb89 already unused, calling callback (000000002988a2da) 00010000:00010000:1.0:1781047573.227832:0:19809:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227848:0:19809:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227855:0:19809:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227860:0:19809:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1781047573.227864:0:19809:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.227870:0:19809:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000004dd5eb89/0x75014a414d98e130 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19809 timeout: 0 lvb_type: 0 00000002:00100000:1.0:1781047573.227886:0:19809:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@00000000a161b523 x1867562513309440/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 592/640 e 0 to 0 dl 1781047963 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'rm.0' 00010000:00010000:1.0:1781047573.227901:0:19809:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:2.0:1781047573.240951:0:19816:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:2.0:1781047573.240971:0:19816:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1781047573.240985:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.240997:0:19816:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:2.0:1781047573.241005:0:19816:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:2.0:1781047573.241007:0:19816:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:2.0:1781047573.241012:0:19816:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 00000000a1c5156f 00010000:00010000:2.0:1781047573.241014:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.241019:0:19816:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1781047573.241030:0:19816:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:2.0:1781047573.241038:0:19816:0:(namei.c:1111:ll_lookup_nd()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), flags=8 00000080:00200000:2.0:1781047573.241042:0:19816:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=getattr 00000002:00010000:2.0:1781047573.241052:0:19816:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1781047573.241098:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.241108:0:19816:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.241214:0:19816:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1781047573.241236:0:19816:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309504:192.168.201.144@tcp:101:lfs.0 00000100:00100000:2.0:1781047573.241302:0:19816:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 388 seconds 00000100:00100000:2.0:1781047573.242714:0:19816:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309504:192.168.201.144@tcp:101:lfs.0 00010000:00010000:2.0:1781047573.242728:0:19816:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242744:0:19816:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242748:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242753:0:19816:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242757:0:19816:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242762:0:19816:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 000000009e4eb824 already unused, calling callback (000000002988a2da) 00010000:00010000:2.0:1781047573.242766:0:19816:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242776:0:19816:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242796:0:19816:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242804:0:19816:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1781047573.242813:0:19816:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242819:0:19816:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e137 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1781047573.242830:0:19816:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=8 disposition=7, status=0 req@00000000523c4981 x1867562513309504/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 584/640 e 0 to 0 dl 1781047963 ref 1 fl Complete:RQU/0/0 rc 301/301 job:'lfs.0' 00010000:00010000:2.0:1781047573.242848:0:19816:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 30 res 8589934599/1 (0 0) 00000080:00200000:2.0:1781047573.242894:0:19816:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:2.0:1781047573.242902:0:19816:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1781047573.242909:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242916:0:19816:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:2.0:1781047573.242922:0:19816:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:2.0:1781047573.242924:0:19816:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:2.0:1781047573.242928:0:19816:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000008623897 00010000:00010000:2.0:1781047573.242930:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242937:0:19816:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1781047573.242955:0:19816:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:2.0:1781047573.242967:0:19816:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:2.0:1781047573.242973:0:19816:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1781047573.242978:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.242985:0:19816:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:2.0:1781047573.242992:0:19816:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:2.0:1781047573.242994:0:19816:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:2.0:1781047573.242997:0:19816:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000013e529ac 00010000:00010000:2.0:1781047573.243000:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.243006:0:19816:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:2.0:1781047573.243012:0:19816:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 3 00000080:00200000:2.0:1781047573.243016:0:19816:0:(namei.c:1183:ll_atomic_open()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), file 00000000176c37cf, open_flags a141, mode 81b6 opened 0 00000080:00200000:2.0:1781047573.243024:0:19816:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=open|creat 00000002:00010000:2.0:1781047573.243037:0:19816:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x200000401:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0120502 00010000:00010000:2.0:1781047573.243072:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.243077:0:19816:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000020:00100000:2.0:1781047573.243093:0:19816:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d7391e43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1781047573.243097:0:19816:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@00000000523c4981 x1867562513309632/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 528/67832 e 0 to 0 dl 0 ref 1 fl New:PQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:2.0:1781047573.243102:0:19816:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000100:00100000:2.0:1781047573.243114:0:19816:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309632:192.168.201.144@tcp:101:lfs.0 00000100:00100000:2.0:1781047573.243742:0:19816:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 388 seconds 00000100:00100000:0.0:1781047573.245751:0:19816:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309632:192.168.201.144@tcp:101:lfs.0 00010000:00010000:0.0:1781047573.245771:0:19816:0:(ldlm_request.c:714:ldlm_cli_enqueue_fini()) ### server returned different mode CR ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245789:0:19816:0:(ldlm_request.c:729:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245803:0:19816:0:(ldlm_request.c:742:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245813:0:19816:0:(ldlm_lock.c:1064:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245820:0:19816:0:(ldlm_request.c:796:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245828:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245832:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245836:0:19816:0:(ldlm_lock.c:909:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1781047573.245842:0:19816:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=3 disposition=8000037, status=0 req@00000000523c4981 x1867562513309632/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 528/600 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'lfs.0' 00000002:00100000:0.0:1781047573.245854:0:19816:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000523c4981 x1867562513309632/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 528/600 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'lfs.0' 00000002:00010000:0.0:1781047573.245861:0:19816:0:(mdc_locks.c:1179:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.245869:0:19816:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 16 res 8589935617/52 (0 0) 00000080:00200000:0.0:1781047573.245881:0:19816:0:(namei.c:1412:ll_create_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=open|creat 00000080:00200000:0.0:1781047573.245908:0:19816:0:(llite_lib.c:2838:ll_read_inode2()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22) 00000080:00200000:0.0:1781047573.245914:0:19816:0:(llite_lib.c:2760:ll_update_inode()) inode=[0x200000401:0x34:0x0], updating i_size 0 00000080:00200000:0.0:1781047573.245946:0:19816:0:(file.c:5870:ll_layout_conf()) [0x200000401:0x34:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1781047573.245950:0:19816:0:(namei.c:1379:ll_create_node()) setting l_ast_data to inode [0x200000401:0x34:0x0](0000000067f13a22) 00000080:00010000:0.0:1781047573.245952:0:19816:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000401:0x34:0x0](0000000067f13a22) for lock 0x75014a414d98e13e 00000080:00200000:0.0:1781047573.245961:0:19816:0:(file.c:837:ll_file_open()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22), flags 120501 00000002:00100000:0.0:1781047573.245972:0:19816:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@00000000523c4981 x1867562513309632/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 528/600 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'lfs.0' 00000080:00010000:0.0:1781047573.246018:0:19816:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e13e from it 000000001b3034ab 00010000:00010000:0.0:1781047573.246020:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.246025:0:19816:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1781047573.246035:0:19816:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22), inode mode 81a4 mask 40 00000080:00200000:0.0:1781047573.246076:0:19816:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22), inode mode 81a4 mask 2 00000080:00200000:0.0:1781047573.246084:0:19816:0:(xattr.c:358:ll_xattr_set()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22), xattr lov 00000002:00010000:0.0:1781047573.246096:0:19816:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000401:0x34:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 040000002 00010000:00010000:0.0:1781047573.246322:0:19816:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.246328:0:19816:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1781047573.246336:0:19816:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d7391e43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1781047573.246339:0:19816:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000002db543f2 x1867562513309760/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 2216/133368 e 0 to 0 dl 0 ref 1 fl New:PQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:0.0:1781047573.246344:0:19816:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1781047573.246362:0:19816:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309760:192.168.201.144@tcp:101:lfs.0 00000100:00100000:0.0:1781047573.246521:0:19816:0:(client.c:2500:ptlrpc_set_wait()) set 00000000d8a4ddeb going to sleep for 388 seconds 00000100:00100000:2.0:1781047573.248211:0:2209:0:(events.c:361:request_in_callback()) peer: 12345-192.168.201.144@tcp (source: 12345-192.168.201.144@tcp) 00000100:00100000:2.0:1781047573.248247:0:10676:0:(service.c:2152:ptlrpc_server_handle_req_in()) got req x1867562522665408 00000100:00100000:2.0:1781047573.248292:0:10676:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.144@tcp, seq: 312 00000100:00100000:2.0:1781047573.248299:0:10676:0:(service.c:2324:ptlrpc_server_handle_request()) Handling RPC req@0000000029b8db81 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:7d9c7ecf-6041-42ed-aef7-34739e7e4960+5:7113:x1867562522665408:12345-192.168.201.144@tcp:104: 00010000:00010000:2.0:1781047573.248311:0:10676:0:(ldlm_lockd.c:2468:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.248362:0:10676:0:(ldlm_lockd.c:2159:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1781047573.248365:0:10676:0:(service.c:2372:ptlrpc_server_handle_request()) Handled RPC req@0000000029b8db81 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:7d9c7ecf-6041-42ed-aef7-34739e7e4960+5:7113:x1867562522665408:12345-192.168.201.144@tcp:104: Request processed in 67us (160us total) trans 0 rc 0/0 00000100:00100000:2.0:1781047573.248370:0:10676:0:(nrs_fifo.c:241:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.144@tcp, seq: 312 00010000:00010000:1.0:1781047573.248386:0:4993:0:(ldlm_lockd.c:2789:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1781047573.248391:0:4993:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.248398:0:4993:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 00000000079ba23a already unused, calling callback (000000002988a2da) 00010000:00010000:1.0:1781047573.248401:0:4993:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.248408:0:4993:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.248416:0:4993:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 3/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000080:00200000:1.0:1781047573.248444:0:4993:0:(vvp_object.c:137:vvp_conf_set()) [0x200000401:0x34:0x0]: losing layout lock 00010000:00010000:1.0:1781047573.248492:0:4993:0:(ldlm_request.c:1279:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 2/0,0 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.248499:0:4993:0:(ldlm_request.c:1325:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1781047573.248526:0:4993:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 1/0,0 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1781047573.248533:0:4993:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000079ba23a/0x75014a414d98e13e lrc: 0/0,0 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x9 rrc: 2 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0x2d3d7e31903fbc8c expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1781047573.248550:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513309824:192.168.201.144@tcp:103:ldlm_bl_01.0 00010000:00010000:1.0:1781047573.248551:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1781047573.248555:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1781047573.248558:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047573.249411:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513309824:192.168.201.144@tcp:103:ldlm_bl_01.0 00000100:00100000:0.0:1781047573.250048:0:19816:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309760:192.168.201.144@tcp:101:lfs.0 00010000:00010000:0.0:1781047573.250063:0:19816:0:(ldlm_request.c:641:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250070:0:19816:0:(ldlm_request.c:562:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250075:0:19816:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250079:0:19816:0:(ldlm_lock.c:879:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250083:0:19816:0:(ldlm_lockd.c:1879:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250086:0:19816:0:(ldlm_lockd.c:1893:ldlm_handle_bl_callback()) Lock 0000000009fc4b51 already unused, calling callback (000000002988a2da) 00010000:00010000:0.0:1781047573.250088:0:19816:0:(ldlm_request.c:1546:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250091:0:19816:0:(ldlm_request.c:1556:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x206400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250102:0:19816:0:(ldlm_request.c:1232:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x20e400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250107:0:19816:0:(ldlm_request.c:1248:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1781047573.250110:0:19816:0:(ldlm_lockd.c:1903:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.250114:0:19816:0:(ldlm_lock.c:216:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e145 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4a0f400000000 nid: local remote: 0x0 expref: -99 pid: 19816 timeout: 0 lvb_type: 0 00000002:00100000:0.0:1781047573.250272:0:19816:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=1 disposition=800002b, status=0 req@000000002db543f2 x1867562513309760/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 2216/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000002:00100000:0.0:1781047573.250286:0:19816:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@000000002db543f2 x1867562513309760/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 2216/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000080:00200000:0.0:1781047573.250401:0:19816:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1781047573.250407:0:19816:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0 00000080:00200000:0.0:1781047573.250432:0:19816:0:(llite_lib.c:2760:ll_update_inode()) inode=[0x200000401:0x34:0x0], updating i_size 0 00000002:00100000:0.0:1781047573.250443:0:19816:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@000000002db543f2 x1867562513309760/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 656/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000002:00100000:0.0:1781047573.250456:0:19816:0:(mdc_request.c:944:mdc_close()) @@@ matched open req@000000002db543f2 x1867562513309760/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 656/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 301/301 job:'lfs.0' 00000020:00100000:0.0:1781047573.250484:0:19816:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d7391e43000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1781047573.250488:0:19816:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000009fa84b8b x1867562513309952/t0(0) o35->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781047573.250501:0:19816:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309952:192.168.201.144@tcp:35:lfs.0 00000100:00100000:0.0:1781047573.250537:0:19816:0:(client.c:2500:ptlrpc_set_wait()) set 000000006af0894a going to sleep for 6 seconds 00000100:00100000:0.0:1781047573.251497:0:19816:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513309952:192.168.201.144@tcp:35:lfs.0 00000002:00100000:0.0:1781047573.251515:0:19816:0:(mdc_request.c:854:mdc_free_open()) @@@ free open request, rq_replay=0 req@000000002db543f2 x1867562513309760/t4294967803(4294967803) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 656/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RQU/4/0 rc 301/301 job:'lfs.0' 00000080:00200000:0.0:1781047573.251560:0:19816:0:(file.c:410:ll_file_release()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22) 00000002:00100000:0.0:1781047573.251583:0:19816:0:(mdc_request.c:944:mdc_close()) @@@ matched open req@00000000523c4981 x1867562513309632/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 528/600 e 0 to 0 dl 1781047963 ref 1 fl Complete:RPQU/4/ffffffff rc 0/-1 job:'lfs.0' 00000020:00100000:0.0:1781047573.251599:0:19816:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d7391e43000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1781047573.251603:0:19816:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000005b954604 x1867562513310080/t0(0) o35->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:23/10 lens 392/2192 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781047573.251615:0:19816:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513310080:192.168.201.144@tcp:35:lfs.0 00000100:00100000:0.0:1781047573.251657:0:19816:0:(client.c:2500:ptlrpc_set_wait()) set 000000006af0894a going to sleep for 6 seconds 00000100:00100000:0.0:1781047573.252604:0:19816:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job lfs:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19816:1867562513310080:192.168.201.144@tcp:35:lfs.0 00000002:00100000:0.0:1781047573.252620:0:19816:0:(mdc_request.c:854:mdc_free_open()) @@@ free open request, rq_replay=0 req@00000000523c4981 x1867562513309632/t4294967802(4294967802) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 528/600 e 0 to 0 dl 1781047963 ref 1 fl Complete:RQU/4/ffffffff rc 0/-1 job:'lfs.0' 00000080:00200000:0.0:1781047573.257757:0:19821:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:0.0:1781047573.257778:0:19821:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1781047573.257794:0:19821:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.257807:0:19821:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1781047573.257818:0:19821:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:0.0:1781047573.257821:0:19821:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:0.0:1781047573.257827:0:19821:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000051f2d411 00010000:00010000:0.0:1781047573.257830:0:19821:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.257845:0:19821:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1781047573.257859:0:19821:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 1 00000080:00200000:0.0:1781047573.257872:0:19821:0:(file.c:5227:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc),name=/ 00000002:00010000:0.0:1781047573.257877:0:19821:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1781047573.257883:0:19821:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.257891:0:19821:0:(ldlm_lock.c:1494:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00010000:0.0:1781047573.257898:0:19821:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) 00000080:00010000:0.0:1781047573.257900:0:19821:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](00000000b5d56edc) for lock 0x75014a414d98e0d5 00000080:00010000:0.0:1781047573.257903:0:19821:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e0d5 from it 0000000023391a21 00010000:00010000:0.0:1781047573.257905:0:19821:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.257912:0:19821:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 00000000d65d3025/0x75014a414d98e0d5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbb51 expref: -99 pid: 18036 timeout: 0 lvb_type: 0 00000080:00200000:0.0:1781047573.257919:0:19821:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](00000000b5d56edc), inode mode 41ed mask 3 00000080:00200000:0.0:1781047573.257923:0:19821:0:(namei.c:1183:ll_atomic_open()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), file 00000000a5154a52, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1781047573.257932:0:19821:0:(namei.c:833:ll_lookup_it()) VFS Op:name=f32.sanity-flr, dir=[0x200000007:0x1:0x0](00000000b5d56edc), intent=open|creat 00000002:00010000:0.0:1781047573.257948:0:19821:0:(mdc_locks.c:1328:mdc_intent_lock()) (name: f32.sanity-flr,[0x200000401:0x35:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1781047573.257998:0:19821:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1781047573.258014:0:19821:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00000020:00100000:0.0:1781047573.258025:0:19821:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d7391e43000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1781047573.258036:0:19821:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000003ef3e6f4 x1867562513310208/t0(0) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 2224/67832 e 0 to 0 dl 0 ref 1 fl New:PQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:0.0:1781047573.258043:0:19821:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1781047573.258060:0:19821:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ef3e6f4 pname:cluuid:pid:xid:nid:opc:job dd:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19821:1867562513310208:192.168.201.144@tcp:101:dd.0 00000100:00100000:0.0:1781047573.258207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 000000006af0894a going to sleep for 388 seconds 00000100:00100000:2.0:1781047573.262233:0:19821:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ef3e6f4 pname:cluuid:pid:xid:nid:opc:job dd:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19821:1867562513310208:192.168.201.144@tcp:101:dd.0 00010000:00010000:2.0:1781047573.262254:0:19821:0:(ldlm_request.c:714:ldlm_cli_enqueue_fini()) ### server returned different mode CR ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262274:0:19821:0:(ldlm_request.c:729:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262292:0:19821:0:(ldlm_request.c:742:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 4/0,1 mode: --/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262303:0:19821:0:(ldlm_lock.c:1064:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262314:0:19821:0:(ldlm_request.c:796:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 4/0,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262326:0:19821:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262333:0:19821:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 5/1,1 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00010000:00010000:2.0:1781047573.262340:0:19821:0:(ldlm_lock.c:909:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 4/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00000002:00100000:2.0:1781047573.262351:0:19821:0:(mdc_locks.c:743:mdc_finish_enqueue()) @@@ op=3 disposition=2b, status=0 req@000000003ef3e6f4 x1867562513310208/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 2224/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'dd.0' 00000002:00100000:2.0:1781047573.262362:0:19821:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@000000003ef3e6f4 x1867562513310208/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 2224/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'dd.0' 00000002:00010000:2.0:1781047573.262374:0:19821:0:(mdc_locks.c:847:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 136 ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 0 00000002:00010000:2.0:1781047573.262387:0:19821:0:(mdc_locks.c:1179:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1781047573.262407:0:19821:0:(ldlm_lock.c:1511:ldlm_lock_match_with_skip()) ### not matched ns 0000000097dacabd type 13 mode 16 res 8589935617/52 (0 0) 00000080:00200000:2.0:1781047573.262446:0:19821:0:(namei.c:157:ll_iget()) got inode: [0x200000401:0x34:0x0](0000000067f13a22): rc = 0 00000080:00200000:2.0:1781047573.262459:0:19821:0:(file.c:5870:ll_layout_conf()) [0x200000401:0x34:0x0]: layout version change: 4294967294 -> 1 00000080:00010000:2.0:1781047573.262479:0:19821:0:(llite_internal.h:1711:ll_set_lock_data()) setting l_data to inode [0x200000401:0x34:0x0](0000000067f13a22) for lock 0x75014a414d98e14c 00000080:00010000:2.0:1781047573.262489:0:19821:0:(dcache.c:293:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x34:0x0](0000000067f13a22) 00000080:00200000:2.0:1781047573.262500:0:19821:0:(file.c:837:ll_file_open()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22), flags 101101 00000002:00100000:2.0:1781047573.262516:0:19821:0:(mdc_request.c:833:mdc_set_open_replay_data()) @@@ Set up open replay data req@000000003ef3e6f4 x1867562513310208/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 664/736 e 0 to 0 dl 1781047963 ref 2 fl Complete:RPQU/4/0 rc 0/0 job:'dd.0' 00000080:00010000:2.0:1781047573.262581:0:19821:0:(dcache.c:193:ll_intent_drop_lock()) releasing lock with cookie 0x75014a414d98e14c from it 0000000090412049 00010000:00010000:2.0:1781047573.262585:0:19821:0:(ldlm_lock.c:820:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 3/1,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 3 00010000:00010000:2.0:1781047573.262593:0:19821:0:(ldlm_lock.c:902:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9d7391e43000 lock: 0000000009fc4b51/0x75014a414d98e14c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x34:0x0].0x0 bits 0x9/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x2d3d7e31903fbcc4 expref: -99 pid: 19821 timeout: 0 lvb_type: 3 00000080:00200000:2.0:1781047573.262608:0:19821:0:(file.c:5661:ll_inode_permission()) VFS Op:inode=[0x200000401:0x34:0x0](0000000067f13a22), inode mode 81a4 mask 42 00000080:00200000:2.0:1781047573.262616:0:19821:0:(llite_lib.c:2191:ll_setattr_raw()) lustre: setattr inode [0x200000401:0x34:0x0](0000000067f13a22) from 0 to 0, valid a068, hsm_import 0 00000020:00100000:2.0:1781047573.262652:0:19821:0:(genops.c:2339:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9d7391e43000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1781047573.262659:0:19821:0:(client.c:742:ptlrpc_reassign_next_xid()) @@@ reassign xid req@000000000e018bdc x1867562513310336/t0(0) o36->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781047573.262679:0:19821:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job dd:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19821:1867562513310336:192.168.201.144@tcp:36:dd.0 00000100:00100000:2.0:1781047573.262741:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 388 seconds 00000100:00100000:2.0:1781047573.264443:0:19821:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job dd:7d9c7ecf-6041-42ed-aef7-34739e7e4960:19821:1867562513310336:192.168.201.144@tcp:36:dd.0 00000080:00200000:2.0:1781047573.264493:0:19821:0:(file.c:1543:ll_io_set_mirror()) f32.sanity-flr: desiginated mirror: 0 00000080:00200000:2.0:1781047573.264499:0:19821:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x34:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00020000:00200000:2.0:1781047573.264521:0:19821:0:(lov_io.c:825:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1781047573.264534:0:19821:0:(vvp_io.c:222:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1781047573.264547:0:19821:0:(lustre_osc.h:181:osc_enq2ldlm_flags()) flags: 4 00000008:00010000:2.0:1781047573.264551:0:19821:0:(osc_lock.c:1241:osc_lock_init()) ### lock 00000000fda1cbea, osc lock 00000000ebbf5508, flags 0x80000000 00010000:00010000:2.0:1781047573.264579:0:19821:0:(ldlm_lock.c:766:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e153 lrc: 3/0,1 mode: --/PW res: [0x5b:0x0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 19821 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1781047573.264589:0:19821:0:(ldlm_request.c:1026:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x80000000 ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e153 lrc: 3/0,1 mode: --/PW res: [0x5b:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19821 timeout: 0 lvb_type: 1 00010000:00010000:2.0:1781047573.264613:0:19821:0:(ldlm_request.c:1114:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0000-osc-ffff9d7391e43000 lock: 000000009e4eb824/0x75014a414d98e153 lrc: 3/0,1 mode: --/PW res: [0x5b:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 19821 timeout: 0 lvb_type: 1 00000100:00080000:2.0:1781047573.264630:0:19821:0:(client.c:1659:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@00000000b3f605c6 x1867562513310400/t0(0) o101->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/0/ffffffff rc 0/-1 job:'dd.0' 00000100:00100000:2.0:1781047573.264639:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047574.017320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047574.273248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047575.041322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047575.297342:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047575.425342:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047575.425350:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047575.425353:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047575.425363:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1211 00000100:00080000:2.0:1781047575.425372:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781047575.425376:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513310464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047575.425380:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047575.425391:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047575.425408:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513310592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047575.425451:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047575.425469:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047575.425473:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513310592:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781047575.425476:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781047575.425485:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513310528:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047575.426879:0:2216:0:(client.c:2868:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9d7391e43000: committing for last_committed 4294967807 gen 1 00000100:00080000:0.0:1781047575.426951:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047575.426955:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047575.426960:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513310592:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047575.427037:0:2216:0:(client.c:2899:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@000000003ef3e6f4 x1867562513310208/t4294967806(4294967806) o101->lustre-MDT0000-mdc-ffff9d7391e43000@192.168.201.144@tcp:12/10 lens 664/736 e 0 to 0 dl 1781047963 ref 1 fl Complete:RPQU/4/ffffffff rc 0/-1 job:'dd.0' 00000100:00100000:2.0:1781047575.427051:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513310528:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047575.427061:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513310464:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781047576.065305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047576.321273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047577.089585:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047577.345233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047578.113294:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047578.369250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047579.137276:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047579.394491:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047580.161322:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047580.417262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047580.545370:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047580.545379:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047580.545384:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047580.545398:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1216 00000100:00080000:0.0:1781047580.545402:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781047580.545410:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781047580.545411:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513310656:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781047580.545442:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047580.545461:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513310784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047580.545489:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047580.545493:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047580.545521:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513310720:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781047580.545540:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513310784:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781047580.547120:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513310656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781047580.547174:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513310720:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781047580.547259:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047580.547264:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047580.547270:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513310784:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781047581.185368:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047581.441237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047582.209329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047582.465242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047583.233368:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047583.489245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047584.257341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047584.513212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047585.282293:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047585.537314:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047585.665389:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047585.665399:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047585.665402:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047585.665416:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1221 00000100:00080000:2.0:1781047585.665439:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781047585.665447:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047585.665462:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047585.665467:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513310912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047585.665483:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513310976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047585.665507:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047585.665511:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047585.665558:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513310848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047585.665682:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513310976:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047585.666914:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513310912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047585.667054:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513310848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047585.667215:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047585.667220:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047585.667224:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513310976:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047586.305361:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047586.561284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047587.329399:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047587.585233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047588.354260:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047588.609227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047589.377301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047589.633214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047590.401320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047590.657247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047590.785327:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047590.785332:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047590.785335:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047590.785342:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1226 00000100:00080000:2.0:1781047590.785345:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781047590.785349:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047590.785357:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047590.785367:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000523c4981 x1867562513311168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047590.785396:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047590.785399:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047590.785418:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513311104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047590.785450:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311168:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047590.785486:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047590.785492:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513311040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047590.787184:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513311104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047590.787213:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513311040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047590.787276:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047590.787280:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047590.787284:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311168:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047591.425333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047591.681275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047592.449322:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047592.705271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047593.473396:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047593.729270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047594.497320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047594.753267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047595.521289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047595.777275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047595.905400:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047595.905427:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047595.905431:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781047595.905436:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513311232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047595.905440:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1231 00000100:00080000:1.0:1781047595.905443:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781047595.905450:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047595.905462:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047595.905480:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513311360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781047595.905532:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047595.905561:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781047595.905568:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513311296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047595.905579:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047595.905585:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047595.906988:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513311296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047595.907340:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513311232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047595.913610:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047595.913623:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047595.913635:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311360:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047596.545386:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047596.801211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047597.569299:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047597.825289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047598.593349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047598.849253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047599.617356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047599.873271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047600.641343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047600.897226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047601.025387:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781047601.025417:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513311424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047601.025429:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047601.025434:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047601.025439:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1236 00000100:00080000:2.0:1781047601.025442:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781047601.025448:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047601.025460:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047601.025479:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084e7220c x1867562513311552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047601.025505:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047601.025509:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047601.025530:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781047601.025536:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513311488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047601.025547:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311552:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047601.027302:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513311488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047601.027306:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047601.027312:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047601.027317:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311552:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047601.027324:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513311424:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781047601.665376:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047601.921302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047602.689423:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047602.945296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047603.713369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047603.969217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000080:00200000:1.0:1781047604.097294:0:38:0:(vvp_io.c:1820:vvp_io_init()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 2 restore needed 0 00020000:00200000:1.0:1781047604.097313:0:38:0:(lov_io.c:825:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1781047604.097340:0:38:0:(vvp_io.c:313:vvp_io_fini()) [0x200000401:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0 00010000:00010000:0.0:1781047604.737341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047604.993249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047605.761332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047606.017979:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047606.145334:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047606.145344:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047606.145348:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047606.145361:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1242 00000100:00100000:0.0:1781047606.145362:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513311616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047606.145365:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047606.145376:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047606.145388:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047606.145405:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000000dbb310 x1867562513311744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047606.145444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047606.145447:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047606.145468:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513311680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047606.145476:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311744:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047606.147161:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513311616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047606.147178:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513311680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047606.147227:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047606.147231:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047606.147235:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311744:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047606.785406:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047607.041265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047607.809353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047608.065281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047608.833396:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047609.089273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047609.857329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047610.113264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047610.881349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047611.137290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047611.265338:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047611.265347:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047611.265351:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781047611.265358:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513311808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047611.265361:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1247 00000100:00080000:1.0:1781047611.265367:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047611.265373:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047611.265384:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047611.265399:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000013bcd30c x1867562513311936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047611.265414:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047611.265417:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047611.265430:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513311872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047611.265466:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311936:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047611.266978:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513311808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047611.266996:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513311872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047611.267040:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047611.267046:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047611.267050:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513311936:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047611.905316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047612.161214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047612.929350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047613.185255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047613.953272:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047614.209278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047614.977559:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047615.233254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047616.001334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047616.257255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047616.385400:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047616.385409:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047616.385413:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781047616.385433:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513312000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047616.385443:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1252 00000100:00080000:0.0:1781047616.385448:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047616.385455:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047616.385469:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047616.385487:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513312128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047616.385509:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047616.385513:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047616.385551:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781047616.385559:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312128:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047616.385561:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513312064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047616.387725:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513312064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047616.387742:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781047616.387747:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513312000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047616.387749:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047616.387757:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312128:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047617.025319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047617.281283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047618.049340:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047618.305207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047619.073267:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047619.329274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047620.097357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047620.353253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047621.121349:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047621.377226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047621.505445:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047621.505455:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047621.505459:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047621.505472:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1257 00000100:00080000:1.0:1781047621.505476:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047621.505483:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047621.505494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047621.505517:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ce8e0ca x1867562513312320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047621.505541:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047621.505545:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047621.505582:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513312256:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047621.505651:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312320:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047621.505709:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781047621.505721:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513312192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047621.506970:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513312256:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047621.507022:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513312192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047621.507064:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047621.507072:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047621.507079:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312320:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047622.145372:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047622.401228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047623.169303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047623.425188:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047624.193316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047624.449267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047625.217341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047625.473245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047626.241334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047626.497256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047626.625351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781047626.625352:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513312384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047626.625362:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047626.625367:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047626.625374:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1262 00000100:00080000:1.0:1781047626.625377:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047626.625385:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047626.625404:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047626.625423:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513312512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047626.625447:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047626.625451:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047626.625475:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513312448:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047626.625494:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312512:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047626.628157:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513312384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047626.628169:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781047626.628171:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513312448:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047626.628176:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047626.628179:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312512:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047627.265322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047627.521258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047628.289350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047628.545272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047629.313342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047629.569252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047630.337315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047630.593217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047631.361312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047631.617276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047631.745641:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047631.745650:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047631.745654:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047631.745665:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1267 00000100:00080000:2.0:1781047631.745668:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047631.745674:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047631.745695:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047631.745709:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513312704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047631.745737:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047631.745741:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047631.745774:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513312640:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047631.745831:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312704:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047631.745920:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047631.745938:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513312576:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047631.747943:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047631.747951:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047631.747960:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312704:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047631.747968:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513312576:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047631.748003:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513312640:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781047632.385355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047632.641291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047633.409359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047633.665289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047634.433354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047634.689266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047635.457315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047635.713273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047636.481355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047636.737272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047636.865376:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047636.865384:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047636.865387:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047636.865399:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1272 00000100:00080000:2.0:1781047636.865414:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047636.865421:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047636.865432:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047636.865446:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513312896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047636.865461:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047636.865464:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047636.865491:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513312832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047636.865509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312896:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047636.865617:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513312768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047636.867070:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513312768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:0.0:1781047636.867093:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00100000:3.0:1781047636.867114:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513312832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047636.870232:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047636.870243:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047636.870254:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513312896:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047637.505420:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047637.761235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047638.529349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047638.785254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047639.553393:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047639.809208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047640.577369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047640.833238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047641.601358:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047641.857233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047641.985342:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047641.985349:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047641.985351:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047641.985360:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1277 00000100:00080000:2.0:1781047641.985364:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047641.985369:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047641.985378:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047641.985392:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018a334bb x1867562513313088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047641.985415:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047641.985419:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047641.985474:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313088:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781047641.985504:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513312960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047641.985574:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781047641.985582:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047641.988679:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513312960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047641.988707:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047641.988712:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047641.988715:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047641.988719:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313088:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047642.625336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047642.881227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047643.649342:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047643.905258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047644.673319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047644.929256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047645.697361:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047645.953273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047646.721357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047646.977251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047647.105299:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047647.105304:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047647.105307:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047647.105315:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1283 00000100:00080000:1.0:1781047647.105317:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047647.105321:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047647.105329:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047647.105341:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d7e018b1 x1867562513313280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047647.105358:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047647.105362:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047647.105381:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513313152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047647.105385:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513313216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047647.105393:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313280:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047647.106924:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513313216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047647.106965:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781047647.106970:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513313152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047647.106971:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047647.106978:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313280:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047647.745299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047648.001253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047648.769321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047649.025246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047649.793373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047650.049777:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047650.817304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047651.073267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047651.841315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047652.097455:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047652.225356:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047652.225365:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047652.225369:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047652.225383:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1288 00000100:00080000:1.0:1781047652.225387:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781047652.225390:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513313344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047652.225398:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047652.225411:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047652.225430:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001cbc13b4 x1867562513313472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047652.225481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047652.225485:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047652.225503:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781047652.225509:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513313408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047652.225941:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313472:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047652.226791:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513313344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047652.226890:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513313408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047652.226920:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047652.226925:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047652.226930:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313472:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047652.865300:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047653.121253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047653.889304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047654.145221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781047654.273351:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313536:192.168.201.144@tcp:17:kworker/2:1.0 00000100:00100000:2.0:1781047654.274807:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313536:192.168.201.144@tcp:17:kworker/2:1.0 00010000:00010000:3.0:1781047654.913358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047655.169235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047655.937268:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047656.193309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047656.961399:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047657.217211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047657.345311:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047657.345318:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047657.345322:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047657.345331:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1293 00000100:00080000:2.0:1781047657.345334:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047657.345354:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047657.345364:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047657.345380:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513313728/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781047657.345382:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513313600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047657.345404:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047657.345407:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047657.345425:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047657.345441:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313728:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047657.346674:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513313600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047657.346723:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047657.346732:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047657.346736:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047657.346740:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313728:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047657.985268:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047658.241237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047659.009501:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047659.265252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047660.033300:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047660.289233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047661.057313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047661.313258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047662.081363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047662.337287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047662.465344:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047662.465352:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047662.465357:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047662.465370:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1298 00000100:00080000:2.0:1781047662.465375:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047662.465382:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047662.465394:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047662.465396:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513313792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047662.465414:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4afdb11 x1867562513313920/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047662.465438:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047662.465460:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047662.465503:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047662.465533:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313920:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047662.466766:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513313792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047662.466804:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513313856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047662.466924:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047662.466930:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047662.466937:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513313920:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047663.105371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047663.361277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047664.129383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047664.385273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047665.153380:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047665.409292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047666.177311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047666.433247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047667.201321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047667.457266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047667.585324:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047667.585330:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047667.585333:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047667.585342:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1303 00000100:00080000:3.0:1781047667.585345:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781047667.585350:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513313984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047667.585352:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047667.585366:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047667.585384:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513314112/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047667.585405:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047667.585408:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047667.585430:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047667.585436:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513314048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047667.585485:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314112:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781047667.586859:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781047667.586861:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513314048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047667.586867:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047667.586874:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314112:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047667.586880:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513313984:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047668.225370:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047668.481267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047669.249335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047669.505257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047670.273274:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047670.529269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047671.297334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047671.553280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047672.321370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047672.577258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047672.705409:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047672.705416:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047672.705420:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047672.705435:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1308 00000100:00080000:3.0:1781047672.705453:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047672.705459:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047672.705470:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047672.705484:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513314304/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047672.705504:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047672.705506:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047672.705602:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314304:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047672.705774:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047672.705849:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781047672.705856:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513314176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047672.707473:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047672.707483:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047672.707492:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314304:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047672.707517:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513314176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047672.707544:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314240:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047673.345316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047673.601244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047674.369318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047674.625231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047675.393264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047675.649237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047676.417280:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047676.673217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047677.441319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047677.697248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047677.825345:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047677.825355:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047677.825358:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047677.825372:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1313 00000100:00080000:3.0:1781047677.825377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047677.825385:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047677.825398:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047677.825418:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513314496/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781047677.825424:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047677.825452:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047677.825457:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047677.825502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314496:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047677.825525:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047677.826978:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047677.827013:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047677.827022:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047677.827024:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047677.827030:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314496:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047678.465319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047678.721266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047679.489354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047679.745229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047680.513347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047680.769260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047681.537343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047681.793303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047682.561340:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047682.817267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047682.945343:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047682.945352:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047682.945356:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047682.945369:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1318 00000100:00080000:2.0:1781047682.945372:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047682.945381:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781047682.945384:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314624:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781047682.945393:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047682.945408:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c170795f x1867562513314688/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047682.945432:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047682.945443:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047682.945473:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314688:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047682.945485:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047682.948507:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047682.948509:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047682.948511:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047682.948517:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047682.948533:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314688:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047683.585452:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047683.841267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047684.609351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047684.865237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047685.633325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047685.889258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047686.657373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047686.913288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047687.681380:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047687.937282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047688.065337:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047688.065347:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047688.065351:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047688.065364:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1323 00000100:00080000:2.0:1781047688.065368:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047688.065376:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781047688.065380:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314752:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781047688.065389:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047688.065409:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513314880/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047688.065445:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047688.065449:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047688.065473:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047688.065489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314880:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047688.066834:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047688.066858:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513314816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047688.066885:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047688.066890:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047688.066896:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513314880:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047688.705316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047688.961255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047689.729384:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047689.985260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047690.753360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047691.009261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047691.777276:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047692.033282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047692.801397:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047693.057289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047693.185305:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047693.185315:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047693.185319:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047693.185331:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1329 00000100:00080000:2.0:1781047693.185336:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047693.185352:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781047693.185356:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315008:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781047693.185366:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047693.185391:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513315072/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047693.185412:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047693.185415:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047693.185447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047693.185461:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047693.186775:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047693.186783:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513314944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047693.186807:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047693.186813:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047693.186818:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315072:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047693.825369:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047694.081239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047694.849285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047695.105267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047695.873329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047696.129254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047696.897258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047697.153266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047697.921385:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047698.177294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781047698.305479:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513315136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047698.305481:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047698.305489:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047698.305493:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047698.305499:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1334 00000100:00080000:3.0:1781047698.305502:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047698.305509:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047698.305520:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047698.305539:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000048c6b8a x1867562513315264/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047698.305562:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047698.305565:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047698.305595:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047698.305610:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315264:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047698.306992:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513315136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047698.307293:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047698.307298:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047698.307303:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315264:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047698.307370:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315200:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047698.945362:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047699.201262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047699.969375:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047700.225365:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047700.993334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047701.249232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047702.017360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047702.273267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047703.041382:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047703.297286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047703.425349:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047703.425358:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047703.425362:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781047703.425384:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047703.425389:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1339 00000100:00080000:3.0:1781047703.425394:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047703.425400:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047703.425414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047703.425437:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513315456/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047703.425457:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047703.425461:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047703.425495:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315456:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047703.425497:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781047703.425509:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513315328:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047703.427193:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781047703.427197:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513315328:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047703.427201:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047703.427206:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315456:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047703.427222:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315392:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047704.065332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047704.321271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047705.089343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047705.345277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047706.113326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047706.369254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047707.137265:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047707.393258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047708.161297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047708.417260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047708.545335:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047708.545343:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047708.545347:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047708.545359:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1344 00000100:00080000:3.0:1781047708.545364:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047708.545371:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047708.545383:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513315520:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047708.545384:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047708.545403:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513315648/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047708.545425:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047708.545429:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047708.545470:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047708.545475:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315648:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047708.546763:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047708.546770:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513315520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047708.546788:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047708.546793:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047708.546798:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315648:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047709.185364:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047709.441252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047710.209333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047710.465282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047711.233324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047711.489290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047712.257303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047712.513263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047713.281359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047713.537297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047713.665363:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781047713.665407:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047713.665448:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047713.665453:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047713.665459:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1349 00000100:00080000:3.0:1781047713.665462:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047713.665469:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047713.665483:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047713.665513:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513315840/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047713.665535:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781047713.665536:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1781047713.665540:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047713.665552:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513315712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047713.665570:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315840:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047713.669394:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513315712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047713.669418:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513315776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047713.669419:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047713.669426:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047713.669432:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513315840:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047714.305366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047714.561267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047715.330659:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047715.585315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047716.353988:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047716.610027:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047717.377355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047717.633233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047718.401392:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047718.657200:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047718.785356:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047718.785365:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047718.785370:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047718.785382:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1354 00000100:00100000:2.0:1781047718.785383:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513315904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047718.785389:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047718.785398:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047718.785411:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047718.785430:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513316032/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047718.785456:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047718.785460:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047718.785482:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047718.785487:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513315968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047718.785495:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316032:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047718.787121:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513315968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047718.787169:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513315904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047718.787351:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047718.787355:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047718.787360:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316032:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047719.425335:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047719.681323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047720.449325:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047720.705280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047721.474065:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047721.729278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047722.497329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047722.753347:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047723.521328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047723.777323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047723.905321:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047723.905331:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047723.905352:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047723.905363:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1359 00000100:00100000:2.0:1781047723.905364:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513316096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047723.905367:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047723.905373:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047723.905398:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047723.905415:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513316224/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781047723.905469:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047723.905475:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513316160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047723.905480:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316224:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781047723.905524:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047723.905528:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047723.907168:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513316160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047723.907187:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513316096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047723.907456:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047723.907460:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047723.907463:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316224:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047724.545486:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047724.801260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047725.569310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047725.825283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047726.593349:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047726.849255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047727.617354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047727.873283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047728.641339:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047728.897280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047729.025374:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047729.025382:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047729.025386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781047729.025400:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513316288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047729.025400:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1364 00000100:00080000:1.0:1781047729.025404:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047729.025415:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047729.025428:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047729.025449:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008bfccf5d x1867562513316416/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047729.025491:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047729.025494:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047729.025516:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513316352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047729.025541:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047729.032113:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513316288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047729.032162:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047729.032169:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047729.032175:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047729.032271:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513316352:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047729.665352:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047729.921254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047730.689341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047730.945217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047731.713298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047731.969284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047732.737332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047732.993286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047733.761402:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047734.017276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047734.145343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047734.145353:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047734.145358:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047734.145370:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1370 00000100:00080000:1.0:1781047734.145374:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781047734.145376:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513316480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047734.145384:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047734.145396:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047734.145410:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513316608/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047734.145431:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047734.145435:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047734.145461:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513316544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047734.145479:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316608:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781047734.147248:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047734.147257:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047734.147263:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316608:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047734.147327:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513316544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047734.147357:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513316480:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781047734.785374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047735.041275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047735.809425:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047736.066257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047736.833311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047737.089243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047737.857291:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047738.113205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047738.881300:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047739.137275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047739.265380:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047739.265389:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047739.265393:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047739.265407:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1375 00000100:00080000:3.0:1781047739.265411:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047739.265418:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047739.265425:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513316736:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781047739.265430:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047739.265447:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513316800/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047739.265473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047739.265477:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047739.265515:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316800:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781047739.265523:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513316672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047739.267743:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513316736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047739.267743:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513316672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047739.267767:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047739.267773:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047739.267779:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316800:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047739.906503:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047740.161274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047740.929475:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047741.185294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047741.953361:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047742.209406:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047742.977303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047743.233821:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047744.001263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047744.257271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047744.385354:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047744.385364:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047744.385368:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047744.385382:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1380 00000100:00080000:3.0:1781047744.385386:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047744.385396:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047744.385397:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513316864:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047744.385410:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047744.385429:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513316992/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047744.385475:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047744.385478:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047744.385509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316992:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047744.385523:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047744.385533:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513316928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047744.387009:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513316928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047744.387013:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047744.387019:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047744.387023:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513316992:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047744.387031:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513316864:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047745.025319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047745.281266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047746.049388:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047746.305205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047747.074468:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047747.329265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047748.097348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047748.353282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047749.121314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047749.377261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047749.505311:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047749.505321:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047749.505324:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781047749.505327:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047749.505332:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1385 00000100:00080000:3.0:1781047749.505336:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047749.505342:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047749.505366:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047749.505383:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513317184/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047749.505402:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047749.505419:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047749.505442:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513317120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047749.505456:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317184:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047749.506880:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513317120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047749.506912:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047749.506959:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047749.506964:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047749.506969:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317184:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047750.145298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047750.401242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047751.169282:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047751.425234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047752.193333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047752.449255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047753.217291:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047753.473246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047754.241338:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047754.497241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047754.625346:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047754.625354:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047754.625357:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047754.625367:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1390 00000100:00080000:3.0:1781047754.625371:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047754.625377:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047754.625381:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317248:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781047754.625388:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047754.625475:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bc445f2a x1867562513317376/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781047754.625480:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047754.625489:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513317312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047754.625492:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047754.625496:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047754.625562:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317376:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781047754.626539:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513317312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047754.626547:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047754.626743:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047754.626748:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047754.626753:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317376:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047755.265326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047755.521256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047756.289333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047756.545254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047757.313318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047757.569251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047758.337318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047758.593255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047759.361319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047759.617276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781047759.745366:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513317440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047759.745366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047759.745375:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047759.745380:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047759.745387:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1395 00000100:00080000:1.0:1781047759.745390:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047759.745397:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047759.745410:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047759.745436:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b738af86 x1867562513317568/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047759.745460:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047759.745464:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047759.745482:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513317504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047759.745503:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317568:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047759.746993:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513317440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047759.746994:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513317504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047759.747016:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047759.747021:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047759.747027:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317568:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047760.385329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047760.641287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047761.410172:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047761.665251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047762.433330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047762.689273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047763.457281:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047763.713249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047764.481353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047764.737250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047764.865308:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047764.865334:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047764.865338:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047764.865349:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1400 00000100:00080000:3.0:1781047764.865352:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047764.865359:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047764.865371:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047764.865389:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bc445f2a x1867562513317760/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047764.865418:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047764.865422:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047764.865451:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513317696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047764.865463:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317760:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047764.865558:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047764.867278:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513317696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047764.867283:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047764.867289:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047764.867296:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317760:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781047764.867493:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317632:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781047765.505379:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047765.761230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047766.529310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047766.786842:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047767.553365:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047767.810103:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047768.577353:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047768.833252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047769.602176:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047769.857225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047769.985309:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047769.985316:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047769.985319:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047769.985330:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1405 00000100:00080000:3.0:1781047769.985334:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047769.985341:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047769.985352:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047769.985353:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047769.985368:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513317952/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047769.985414:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047769.985418:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047769.985438:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513317888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047769.985568:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317952:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781047769.987120:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513317888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047769.987233:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513317824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047769.987613:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047769.987619:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047769.987624:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513317952:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047770.625305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047770.881223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047771.649271:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047771.905232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047772.673346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047772.931064:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047773.697375:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047773.953244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047774.721326:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047774.977231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047775.105306:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047775.105311:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047775.105314:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047775.105323:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1411 00000100:00080000:0.0:1781047775.105325:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047775.105330:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047775.105338:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047775.105341:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047775.105367:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513318144/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047775.105383:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047775.105386:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047775.105417:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513318080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047775.105433:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318144:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047775.106618:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047775.106630:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513318080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047775.106638:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047775.106644:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047775.106650:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318144:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047775.745302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047776.001231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047776.769311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047777.025256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047777.793298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047778.049241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047778.817298:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047779.073235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047779.841342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047780.097235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047780.225363:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047780.225371:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047780.225374:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047780.225383:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1416 00000100:00080000:3.0:1781047780.225386:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781047780.225388:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513318208:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047780.225392:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047780.225400:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047780.225413:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513318336/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781047780.225463:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318336:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047780.225472:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047780.225476:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513318272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047780.225509:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047780.225513:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047780.226831:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513318272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047780.226838:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047780.226843:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047780.226850:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318336:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047780.226865:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513318208:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781047780.865275:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047781.121245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047781.889359:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047782.145236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047782.913295:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047783.169231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047783.937336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047784.193243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047784.961334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047785.217202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047785.345512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047785.345521:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047785.345525:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047785.345539:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1421 00000100:00080000:0.0:1781047785.345543:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047785.345552:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047785.345566:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047785.345570:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047785.345587:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513318528/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781047785.345654:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318528:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047785.345667:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781047785.345676:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513318464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047785.345767:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047785.345792:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047785.347194:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047785.347224:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513318464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047785.347353:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047785.347359:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047785.347365:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318528:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047785.985348:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047786.241232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047787.009309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047787.265232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047788.033284:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047788.289237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047789.057320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047789.313251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047790.081244:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047790.337260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047790.465301:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047790.465307:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047790.465311:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047790.465321:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1426 00000100:00080000:1.0:1781047790.465325:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781047790.465329:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318592:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047790.465332:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047790.465345:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047790.465369:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a5a8ed39 x1867562513318720/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047790.465406:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047790.465409:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047790.465421:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781047790.465430:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513318656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047790.465437:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318720:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047790.466957:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513318656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:02020000:3.0:1781047790.466957:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00100000:0.0:1781047790.466977:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318592:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047790.470256:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047790.470265:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047790.470275:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318720:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047791.105339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047791.361283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047792.129392:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047792.385228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047793.153317:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047793.409236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047794.177327:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047794.433221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047795.201307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047795.457252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047795.585316:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047795.585324:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047795.585328:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047795.585337:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1431 00000100:00080000:3.0:1781047795.585341:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781047795.585347:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513318848:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047795.585349:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047795.585358:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047795.585386:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513318912/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047795.585408:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047795.585412:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047795.585441:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781047795.585449:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513318784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047795.585459:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318912:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047795.586926:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513318784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047795.586935:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047795.586940:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047795.586947:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513318848:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047795.586948:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513318912:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047796.225310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047796.481258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047797.249307:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047797.505236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047798.273656:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047798.529241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047799.297294:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047799.553306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047800.321316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047800.577216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047800.705333:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047800.705340:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047800.705343:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047800.705352:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1436 00000100:00080000:1.0:1781047800.705356:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047800.705363:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047800.705373:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047800.705374:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318976:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047800.705389:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513319104/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047800.705409:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047800.705412:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047800.705433:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513319040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047800.705461:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319104:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047800.706702:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513318976:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047800.706719:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513319040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047800.706872:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047800.706876:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047800.706881:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319104:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047801.345302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047801.601231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047802.369356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047802.625205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047803.393304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047803.649245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047804.417308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047804.673245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047805.441278:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047805.697258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047805.825293:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047805.825299:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047805.825315:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047805.825324:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1441 00000100:00080000:1.0:1781047805.825327:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781047805.825333:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513319168:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047805.825334:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047805.825346:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047805.825375:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f819707f x1867562513319296/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047805.825395:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047805.825399:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047805.825417:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513319232:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047805.825432:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319296:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047805.827411:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513319232:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047805.827412:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513319168:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047805.828214:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047805.828219:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047805.828224:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319296:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047806.465309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047806.721247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047807.489284:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047807.745237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047808.513296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047808.769252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047809.537349:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047809.793236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047810.562322:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047810.817236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047810.945332:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047810.945339:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047810.945343:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047810.945355:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1446 00000100:00080000:1.0:1781047810.945359:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781047810.945365:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513319360:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047810.945366:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047810.945379:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047810.945397:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1c3efb9 x1867562513319488/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047810.945424:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781047810.945453:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1781047810.945453:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047810.945464:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513319424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047810.945504:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319488:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781047810.947002:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781047810.947008:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513319424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047810.947010:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047810.947028:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513319360:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047810.947030:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319488:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047811.585383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047811.841239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047812.609329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047812.865259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047813.633282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047813.889223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047814.657354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047814.913244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047815.681411:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047815.937282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047816.065390:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047816.065399:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047816.065403:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047816.065414:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1451 00000100:00080000:3.0:1781047816.065417:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047816.065424:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047816.065435:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047816.065442:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513319552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047816.065452:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513319680/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047816.065474:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047816.065477:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047816.065501:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513319616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047816.065526:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319680:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781047816.066718:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781047816.066720:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513319552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047816.066724:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047816.066725:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513319616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047816.066733:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319680:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047816.705367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047816.961259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047817.730794:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047817.985323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047818.753366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047819.009241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047819.778601:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047820.033247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047820.801383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047821.057269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047821.185345:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047821.185354:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047821.185358:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047821.185372:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1457 00000100:00100000:2.0:1781047821.185373:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513319744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047821.185377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047821.185403:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047821.185414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047821.185430:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513319872/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047821.185447:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047821.185449:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047821.185467:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513319808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047821.185489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319872:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047821.186847:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513319808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047821.186847:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513319744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047821.186858:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047821.186870:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047821.186877:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513319872:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047821.825315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047822.081230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047822.849318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047823.105226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047823.873297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047824.129239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047824.897311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047825.153215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047825.921286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047826.177241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047826.305330:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047826.305337:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047826.305340:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047826.305351:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1462 00000100:00080000:1.0:1781047826.305354:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781047826.305359:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513319936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047826.305361:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047826.305372:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047826.305386:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1c3efb9 x1867562513320064/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047826.305406:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047826.305410:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047826.305432:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047826.305453:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320064:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047826.306744:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513319936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047826.306783:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047826.313960:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047826.313974:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047826.313997:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320064:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047826.945297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047827.201243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047827.969334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047828.225263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047828.993263:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047829.249269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047830.017344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047830.273273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047831.041330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047831.297224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047831.425347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047831.425355:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047831.425375:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781047831.425379:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513320128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047831.425385:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1467 00000100:00080000:1.0:1781047831.425390:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047831.425397:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047831.425408:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047831.425425:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513320256/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047831.425445:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047831.425449:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047831.425472:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047831.425502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320256:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047831.426893:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047831.426909:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513320128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047831.426937:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047831.426943:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047831.426948:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320256:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047832.065341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047832.321287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047833.089343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047833.345259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047834.113350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047834.369255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047835.137332:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047835.393261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047836.161347:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047836.417296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047836.545357:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047836.545365:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047836.545370:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047836.545380:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1472 00000100:00080000:0.0:1781047836.545383:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047836.545392:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047836.545403:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047836.545403:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047836.545421:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513320448/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047836.545450:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047836.545453:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047836.545516:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320448:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047836.545518:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781047836.545537:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513320320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047836.546829:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513320320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047836.546854:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047836.546857:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047836.546873:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047836.546879:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320448:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781047837.185374:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047837.441221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047838.209339:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047838.465279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047839.233319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047839.489285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047840.257363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047840.513235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047841.281317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047841.537251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047841.665401:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047841.665408:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047841.665412:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047841.665422:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1477 00000100:00080000:3.0:1781047841.665425:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047841.665431:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047841.665440:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047841.665454:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513320640/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047841.665474:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047841.665477:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047841.665527:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320640:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047841.665580:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513320512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047841.665647:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047841.665659:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513320576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047841.667323:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513320576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047841.667333:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513320512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047841.667425:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047841.667430:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047841.667434:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320640:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047842.305321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047842.561244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047843.329316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047843.585260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047844.353270:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047844.609253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047845.377371:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047845.633246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047846.401354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047846.657277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047846.785404:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047846.785429:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047846.785433:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047846.785446:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1482 00000100:00080000:1.0:1781047846.785450:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047846.785457:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781047846.785467:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513320704:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781047846.785470:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047846.785497:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513320832/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781047846.785564:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320832:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047846.785585:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781047846.785596:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513320768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047846.785652:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047846.785656:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047846.787025:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513320768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047846.787039:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513320704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047846.787107:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047846.787115:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047846.787121:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513320832:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047847.427698:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047847.681312:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047848.449350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047848.705445:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047849.473300:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047849.729266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047850.498627:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047850.753256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047851.521315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047851.777228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047851.905351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047851.905366:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047851.905370:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781047851.905371:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513320896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047851.905379:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1487 00000100:00080000:0.0:1781047851.905382:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047851.905387:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047851.905396:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047851.905413:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018617087 x1867562513321024/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047851.905437:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047851.905440:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047851.905459:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047851.905607:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321024:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047851.906917:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513320960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047851.906962:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047851.906970:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047851.906977:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321024:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047851.907059:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513320896:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781047852.545316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047852.801249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047853.569304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047853.825255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047854.593291:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047854.849250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047855.617313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047855.873227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047856.641317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047856.897287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047857.025281:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047857.025304:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047857.025308:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047857.025315:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1492 00000100:00100000:1.0:1781047857.025315:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513321088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781047857.025321:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047857.025328:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047857.025337:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047857.025354:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513321216/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047857.025377:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047857.025380:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047857.025394:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781047857.025400:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513321152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047857.025426:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321216:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047857.026576:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513321152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047857.026591:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513321088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047857.026626:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047857.026631:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047857.026636:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321216:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047857.665292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047857.921243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047858.689316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047858.945228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047859.713246:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047859.969231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047860.737310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047860.993241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047861.761290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047862.017265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047862.145311:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047862.145317:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047862.145320:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047862.145328:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1498 00000100:00080000:1.0:1781047862.145334:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047862.145340:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047862.145348:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047862.145372:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513321408/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781047862.145378:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513321280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047862.145388:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047862.145392:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047862.145427:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513321344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047862.145427:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321408:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047862.146642:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513321280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047862.146685:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513321344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047862.146698:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047862.146703:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047862.146708:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321408:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047862.785304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047863.041261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047863.809374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047864.065242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047864.833287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047865.089245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047865.857327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047866.113229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047866.881281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047867.137235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047867.265308:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047867.265314:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047867.265316:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047867.265324:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1503 00000100:00080000:3.0:1781047867.265338:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047867.265343:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047867.265352:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513321472:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047867.265352:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047867.265366:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513321600/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047867.265382:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047867.265385:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047867.265411:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321600:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047867.265416:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513321536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047867.266521:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513321472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047867.266539:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781047867.266543:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781047867.266548:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321600:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047867.266552:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513321536:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781047867.905271:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047868.161232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047868.929282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047869.185257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047869.953331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047870.209279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047870.977369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047871.233251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047872.001292:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047872.257250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047872.385352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047872.385356:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047872.385359:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047872.385367:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1508 00000100:00080000:0.0:1781047872.385370:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047872.385375:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047872.385383:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047872.385394:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513321664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047872.385395:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513321792/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047872.385415:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047872.385418:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047872.385460:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513321728:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781047872.385466:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321792:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047872.386975:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513321664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047872.386992:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513321728:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781047872.387028:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047872.387033:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047872.387037:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321792:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047873.025333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047873.281249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047874.049319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047874.305236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047875.073337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047875.329240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047876.097308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047876.353191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047877.121306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047877.377250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047877.505313:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047877.505321:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047877.505324:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047877.505334:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1513 00000100:00080000:3.0:1781047877.505337:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047877.505346:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047877.505347:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513321856:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047877.505367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047877.505381:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513321984/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047877.505401:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047877.505405:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047877.505427:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513321920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047877.505445:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321984:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1781047877.506939:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781047877.506945:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781047877.506950:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513321984:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047877.507458:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513321920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047877.507500:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513321856:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781047878.145294:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047878.401238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047879.169301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047879.425236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047880.193331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047880.449245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047881.217230:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047881.473250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047882.241302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047882.497245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047882.625375:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047882.625390:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047882.625394:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047882.625408:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1518 00000100:00080000:3.0:1781047882.625412:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047882.625419:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047882.625431:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047882.625449:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513322176/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047882.625468:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047882.625470:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047882.625501:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047882.625520:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322176:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047882.625675:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047882.627088:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047882.627184:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047882.627193:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781047882.627199:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781047882.627209:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322176:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047883.265393:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047883.521240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047884.289321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047884.545258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047885.313335:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047885.569226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047886.337342:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047886.593257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047887.361326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047887.617272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047887.745339:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047887.745348:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047887.745352:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047887.745362:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1523 00000100:00080000:3.0:1781047887.745365:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047887.745372:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047887.745378:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322240:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047887.745383:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047887.745403:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513322368/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047887.745426:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047887.745430:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047887.745468:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322368:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047887.745476:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322304:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047887.747267:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047887.747281:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322304:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047887.747291:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781047887.747296:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781047887.747302:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322368:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047888.386308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047888.641258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047889.409407:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047889.665272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047890.433348:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047890.689281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047891.457987:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047891.713252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047892.481315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047892.737231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047892.865317:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047892.865322:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047892.865325:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047892.865332:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1528 00000100:00080000:3.0:1781047892.865335:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047892.865341:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047892.865352:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322432:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047892.865364:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047892.865379:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513322560/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047892.865394:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047892.865397:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047892.865415:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322560:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047892.865419:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047892.866574:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047892.866609:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781047892.866619:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781047892.866625:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781047892.866647:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322560:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047893.505321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047893.761242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047894.529277:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047894.785241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047895.553298:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047895.809243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047896.577318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047896.833276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047897.603201:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047897.857284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047897.985347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047897.985356:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047897.985360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047897.985373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1533 00000100:00080000:3.0:1781047897.985377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781047897.985380:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047897.985386:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047897.985400:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047897.985419:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513322752/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047897.985445:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047897.985449:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047897.985492:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047897.985503:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322752:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047897.988038:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513322624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781047897.988038:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513322688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047897.988065:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047897.988071:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047897.988077:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322752:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781047898.625317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047898.881249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047899.649353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047899.905293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047900.673368:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047900.929260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047901.697272:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047901.953253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047902.721364:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047902.977263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047903.105366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047903.105375:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047903.105379:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781047903.105391:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513322816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047903.105392:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1539 00000100:00080000:1.0:1781047903.105397:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047903.105405:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047903.105424:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047903.105440:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b98b0b8e x1867562513322944/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047903.105461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047903.105465:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047903.105490:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513322880:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047903.105510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322944:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781047903.106843:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513322880:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781047903.106852:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513322816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047903.106933:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047903.106937:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047903.106941:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513322944:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781047903.745291:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047904.001252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781047904.769332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047905.025235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047905.793370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047906.049251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047906.817300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047907.073273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047907.841316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047908.097391:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047908.225336:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047908.225344:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047908.225348:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047908.225360:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1544 00000100:00100000:0.0:1781047908.225361:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781047908.225364:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047908.225399:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047908.225412:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047908.225427:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513323136/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047908.225447:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047908.225451:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047908.225474:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513323072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781047908.225495:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323136:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781047908.226931:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781047908.226934:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513323072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781047908.226959:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047908.226965:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047908.226971:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323136:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781047908.865307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047909.122764:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047909.889708:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047910.145247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047910.913236:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047911.169262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047911.938536:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781047912.193257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047912.961325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047913.217233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047913.346438:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047913.346460:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047913.346464:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047913.346480:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1549 00000100:00080000:3.0:1781047913.346484:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047913.346491:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047913.346503:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513323200:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047913.346506:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047913.346530:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513323328/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047913.346570:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047913.346575:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047913.346608:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513323264:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047913.346631:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323328:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047913.348455:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513323200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047913.348636:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047913.348642:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047913.348665:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323328:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047913.348756:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513323264:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781047913.985314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047914.241271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047915.009315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047915.265243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047916.033338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047916.289235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047917.057294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047917.313247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047918.081298:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047918.337223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047918.465331:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047918.465338:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047918.465342:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047918.465352:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1554 00000100:00080000:0.0:1781047918.465367:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047918.465375:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047918.465385:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047918.465402:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513323520/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047918.465431:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047918.465435:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047918.465520:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323520:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047918.465779:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047918.465815:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781047918.465819:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513323456:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047918.466885:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513323456:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047918.466900:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047918.466949:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047918.466954:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047918.466959:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323520:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047919.105295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047919.361255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047920.129370:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047920.385253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047921.153402:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047921.409265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047922.177350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047922.433302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047923.201329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047923.457285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047923.585494:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047923.585504:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047923.585508:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047923.585522:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1559 00000100:00080000:3.0:1781047923.585527:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781047923.585536:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781047923.585541:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513323648:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781047923.585553:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047923.585587:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513323712/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781047923.585651:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781047923.585656:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323712:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781047923.585661:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513323584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047923.585750:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047923.585755:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047923.587062:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513323584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047923.587093:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513323648:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047923.587105:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047923.587111:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047923.587120:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323712:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047924.225356:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047924.481244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047925.249288:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047925.505261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047926.273457:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047926.529270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047927.297334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047927.553285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047928.321471:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047928.577256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047928.705328:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047928.705335:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781047928.705353:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047928.705355:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047928.705363:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1564 00000100:00080000:0.0:1781047928.705366:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047928.705372:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047928.705382:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047928.705401:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513323904/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047928.705425:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047928.705427:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047928.705445:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513323840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047928.705479:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323904:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047928.706706:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047928.706733:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513323840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047928.707155:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047928.707167:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047928.707171:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513323904:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047929.345304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047929.601272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047930.369352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047930.625242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047931.393305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047931.649262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047932.417319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047932.673243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047933.441290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047933.697234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047933.825330:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047933.825339:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047933.825343:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047933.825356:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1569 00000100:00080000:0.0:1781047933.825359:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047933.825369:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047933.825379:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047933.825394:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b24224a5 x1867562513324096/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047933.825418:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047933.825438:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047933.825468:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047933.825471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324096:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047933.825557:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047933.828280:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047933.828288:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513323968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047933.828314:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047933.828319:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047933.828324:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324096:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047934.465300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047934.721251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047935.489319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047935.745233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047936.513315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047936.769304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047937.537272:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047937.793241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047938.561323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047938.817246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047938.945314:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047938.945320:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047938.945323:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047938.945333:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1574 00000100:00080000:0.0:1781047938.945336:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047938.945344:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047938.945355:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047938.945359:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047938.945375:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513324288/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047938.945404:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047938.945408:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047938.945429:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047938.945451:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324288:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047938.947507:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047938.947531:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047938.947533:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047938.947540:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047938.947552:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324288:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047939.585311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047939.841238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047940.609313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047940.865245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047941.633385:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047941.889248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047942.657396:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047942.913278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047943.682191:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047943.937254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047944.065308:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047944.065316:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047944.065332:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781047944.065335:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047944.065341:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1579 00000100:00080000:0.0:1781047944.065345:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047944.065351:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047944.065361:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047944.065379:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513324480/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781047944.065420:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781047944.065425:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513324416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047944.065462:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047944.065467:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047944.065547:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324480:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047944.067035:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513324416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047944.067044:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:2.0:1781047944.067289:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:2.0:1781047944.070366:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047944.070376:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047944.070385:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324480:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047944.705290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047944.961235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047945.729309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047945.985257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047946.753341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047947.009246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047947.777322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047948.033273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047948.801367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047949.057284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047949.185321:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047949.185328:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047949.185332:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047949.185342:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1585 00000100:00100000:0.0:1781047949.185343:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324544:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781047949.185346:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047949.185355:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047949.185365:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047949.185382:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513324672/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047949.185406:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047949.185409:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047949.185429:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324608:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781047949.185500:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324672:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781047949.186801:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324608:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781047949.186861:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324544:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781047949.186886:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047949.186891:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047949.186896:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324672:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781047949.825321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047950.081238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047950.849323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047951.105280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047951.873342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047952.129198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047952.897277:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047953.153269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047953.921295:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047954.177277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781047954.305372:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047954.305380:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781047954.305385:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781047954.305398:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1590 00000100:00080000:3.0:1781047954.305403:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781047954.305406:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513324736:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781047954.305413:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781047954.305426:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781047954.305445:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513324864/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781047954.305469:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781047954.305473:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047954.305506:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781047954.305522:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513324800:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781047954.305527:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324864:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781047954.306955:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513324800:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781047954.306967:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513324736:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781047954.307021:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047954.307025:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047954.307029:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513324864:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781047954.945264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047955.201255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047955.969597:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047956.225264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047956.993357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047957.249203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047958.017348:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047958.273254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047959.041374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047959.297205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047959.425383:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047959.425408:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047959.425412:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781047959.425414:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324928:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781047959.425419:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1595 00000100:00080000:1.0:1781047959.425423:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047959.425430:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047959.425441:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047959.425455:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513325056/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047959.425489:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047959.425493:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781047959.425515:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324992:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781047959.425539:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325056:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781047959.427327:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513324928:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781047959.427328:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513324992:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781047959.427616:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047959.427621:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047959.427625:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325056:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781047960.065335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047960.321270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047961.089328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047961.345261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047962.113321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047962.369268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047963.137312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047963.393267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047964.161331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047964.417266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047964.545349:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047964.545359:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047964.545363:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047964.545388:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1600 00000100:00100000:1.0:1781047964.545390:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047964.545393:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047964.545403:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047964.545416:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047964.545437:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513325248/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047964.545458:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047964.545461:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047964.545502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325248:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047964.545507:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047964.546936:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047964.546962:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047964.546991:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047964.546995:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047964.547006:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325248:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047965.185251:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047965.441265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047966.209369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047966.465288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047967.233355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047967.489274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047968.257332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047968.513236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047969.281375:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047969.537286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047969.665323:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047969.665330:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047969.665332:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047969.665342:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1605 00000100:00080000:0.0:1781047969.665345:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047969.665352:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781047969.665372:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325376:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781047969.665376:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047969.665395:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513325440/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047969.665416:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047969.665419:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047969.665469:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325312:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047969.665613:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325440:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047969.667396:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047969.667443:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325312:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047969.667509:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047969.667523:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047969.667528:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325440:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047970.305286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047970.561253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047971.329336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047971.585277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047972.353474:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047972.609589:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047973.377374:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047973.633280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047974.401335:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047974.657256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047974.785344:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047974.785353:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047974.785358:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047974.785385:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1610 00000100:00080000:0.0:1781047974.785389:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047974.785399:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781047974.785399:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325504:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781047974.785414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047974.785434:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513325632/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047974.785457:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047974.785461:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047974.785497:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325632:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047974.785505:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781047974.786975:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047974.786980:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047974.786988:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047974.786993:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047974.786995:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325632:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047975.425403:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047975.681275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047976.449347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047976.705225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047977.473372:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047977.729265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047978.497343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047978.753279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047979.521472:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047979.777269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047979.905394:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047979.905404:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047979.905409:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047979.905422:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1615 00000100:00080000:0.0:1781047979.905426:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781047979.905435:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047979.905436:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047979.905449:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047979.905467:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513325824/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047979.905491:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047979.905500:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781047979.905527:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781047979.905540:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325824:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047979.906887:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513325696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047979.906944:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513325760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047979.907057:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781047979.907065:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781047979.907070:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513325824:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781047980.545316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047980.801301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047981.569370:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047981.825318:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047982.593367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047982.849293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047983.617383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047983.873288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047984.641352:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047984.897279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781047985.025477:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781047985.025512:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513325888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781047985.025516:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781047985.025523:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781047985.025536:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1620 00000100:00080000:2.0:1781047985.025543:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781047985.025561:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781047985.025579:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781047985.025603:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2849ba3 x1867562513326016/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781047985.025629:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781047985.025633:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047985.025658:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781047985.025666:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513325952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047985.025880:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781047985.027743:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513325888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047985.027773:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781047985.027780:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047985.027785:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513325952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047985.027787:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326016:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781047985.665383:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047985.921298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047986.689374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047986.945225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047987.713373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047987.969273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047988.737286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047988.993283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047989.761311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047990.017236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781047990.145342:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047990.145349:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781047990.145353:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781047990.145369:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1626 00000100:00100000:1.0:1781047990.145375:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513326080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781047990.145378:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781047990.145390:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781047990.145401:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781047990.145423:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513326208/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781047990.145458:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781047990.145462:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781047990.145490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781047990.145491:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513326144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781047990.146818:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513326144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781047990.146830:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047990.146839:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781047990.146847:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781047990.146920:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513326080:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781047990.785346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047991.041362:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047991.809304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047992.065232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047992.835360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047993.089270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047993.857344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047994.113243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047994.881356:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047995.137260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781047995.265349:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047995.265359:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781047995.265363:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781047995.265376:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1631 00000100:00100000:0.0:1781047995.265381:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513326272:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781047995.265381:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781047995.265391:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781047995.265403:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781047995.265423:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513326400/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781047995.265447:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781047995.265451:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781047995.265486:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326400:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781047995.265501:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513326336:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781047995.267070:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781047995.267080:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781047995.267085:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513326272:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781047995.267087:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326400:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781047995.267181:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513326336:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781047995.905356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047996.161229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047996.929470:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047997.185267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781047997.953335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047998.209262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781047998.977343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781047999.233265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048000.001302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048000.257242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048000.385410:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048000.385417:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048000.385421:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048000.385431:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1636 00000100:00080000:3.0:1781048000.385434:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048000.385438:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513326464:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048000.385442:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048000.385452:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048000.385470:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513326592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048000.385495:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048000.385499:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048000.385530:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781048000.385538:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326592:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048000.385539:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513326528:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048000.386870:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048000.386876:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513326528:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048000.386877:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048000.386885:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326592:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048000.386892:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513326464:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:0.0:1781048001.025346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048001.281258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048002.049328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048002.305258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048003.073332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048003.329254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048004.097314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048004.353225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048005.121316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048005.377290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048005.505343:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048005.505350:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048005.505354:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048005.505364:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1641 00000100:00100000:0.0:1781048005.505377:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513326656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048005.505384:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048005.505392:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048005.505402:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048005.505418:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513326784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048005.505436:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048005.505439:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048005.505457:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513326720:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048005.505471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326784:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048005.506985:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513326656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048005.506989:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513326720:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048005.507034:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048005.507040:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048005.507052:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326784:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048006.145348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048006.401264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048007.169342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048007.425280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048008.193362:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048008.449252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048009.217303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048009.473278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048010.241340:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048010.497285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048010.625370:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048010.625377:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048010.625381:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048010.625394:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513326848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048010.625399:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1646 00000100:00080000:3.0:1781048010.625403:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048010.625409:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048010.625419:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048010.625434:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513326976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048010.625453:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048010.625456:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048010.625482:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513326912:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048010.625493:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326976:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048010.626994:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513326848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048010.626996:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048010.627001:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048010.627004:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513326976:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048010.627042:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513326912:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048011.265283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048011.521252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048012.289317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048012.545271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048013.313332:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048013.569203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048014.337338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048014.593258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048015.361336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048015.617264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048015.745284:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048015.745294:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1781048015.745335:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513327040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048015.745364:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048015.745390:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1651 00000100:00080000:3.0:1781048015.745394:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048015.745400:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048015.745413:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048015.745433:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513327168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781048015.745446:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1781048015.745453:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781048015.745455:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513327104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048015.745459:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048015.745492:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327168:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781048015.746847:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048015.746849:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513327104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048015.746853:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048015.746860:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327168:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048015.746867:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513327040:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781048016.385342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048016.641300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048017.409263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048017.665259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048018.433346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048018.689267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048019.457458:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048019.713256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048020.481328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048020.737266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048020.865377:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048020.865384:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048020.865387:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048020.865398:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1656 00000100:00080000:0.0:1781048020.865401:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781048020.865409:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513327232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048020.865424:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048020.865435:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048020.865449:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002db543f2 x1867562513327360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048020.865464:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048020.865466:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048020.865494:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513327296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048020.865500:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048020.867164:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513327232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048020.867164:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781048020.867172:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513327296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048020.867173:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048020.867182:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327360:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048021.505376:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048021.761282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048022.529360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048022.785271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048023.553330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048023.809286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048024.577345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048024.833270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048025.601406:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048025.857274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048025.985374:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048025.985383:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048025.985388:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048025.985402:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1661 00000100:00100000:1.0:1781048025.985402:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513327424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048025.985408:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048025.985417:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048025.985431:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048025.985448:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dd1a56a6 x1867562513327552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048025.985483:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048025.985486:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048025.985512:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513327488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048025.985525:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327552:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048025.987095:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513327424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048025.987096:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513327488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048025.987100:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048025.987107:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048025.987155:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327552:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048026.625450:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048026.881287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048027.649351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048027.905287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048028.673258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048028.929278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048029.697336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048029.953251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048030.721327:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048030.977253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048031.105356:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048031.105374:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048031.105378:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048031.105390:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1667 00000100:00080000:1.0:1781048031.105394:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781048031.105398:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513327680:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048031.105403:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048031.105415:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048031.105451:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ce8e0ca x1867562513327744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048031.105473:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048031.105477:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048031.105505:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781048031.105512:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513327616:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048031.105568:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327744:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048031.106861:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513327616:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048031.106879:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513327680:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048031.106966:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048031.106973:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048031.106979:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327744:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048031.745303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048032.001231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048032.769277:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048033.025244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048033.793360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048034.049231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048034.817349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048035.073202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048035.841338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048036.097324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048036.225318:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048036.225327:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048036.225332:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048036.225346:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1672 00000100:00100000:2.0:1781048036.225347:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513327808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048036.225351:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048036.225360:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048036.225374:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048036.225394:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513327936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048036.225424:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048036.225428:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048036.225447:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513327872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048036.225468:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327936:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048036.227620:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513327808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048036.227667:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513327872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048036.227751:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048036.227756:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048036.227761:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513327936:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048036.865316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048037.121292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048037.889446:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048038.145278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048038.913286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048039.169284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048039.937358:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048040.193298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048040.961295:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048041.217248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048041.345341:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048041.345352:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048041.345356:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048041.345370:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1677 00000100:00100000:2.0:1781048041.345373:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048041.345375:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048041.345386:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048041.345400:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048041.345419:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513328128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048041.345443:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048041.345446:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048041.345487:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048041.345489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328128:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048041.346944:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048041.346946:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048041.346955:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048041.346961:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048041.346966:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328128:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048041.985358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048042.241292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048043.009341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048043.265274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048044.033350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048044.289259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048045.057384:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048045.313260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048046.081389:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048046.337265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048046.465291:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048046.465297:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048046.465312:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048046.465320:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1682 00000100:00080000:3.0:1781048046.465322:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048046.465327:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048046.465336:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048046.465355:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513328320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781048046.465356:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048046.465382:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048046.465386:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048046.465422:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048046.465439:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328320:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048046.466516:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048046.466584:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048046.466660:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048046.466665:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048046.466670:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328320:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048047.105326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048047.361255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048048.129317:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048048.385239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048049.153330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048049.409235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048050.177303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048050.433237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048051.201304:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048051.457247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048051.585335:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048051.585342:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048051.585346:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048051.585355:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1687 00000100:00080000:3.0:1781048051.585359:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 6s 00000100:00080000:3.0:1781048051.585365:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048051.585375:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048051.585390:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9891000 x1867562513328512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048051.585433:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048051.585436:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048051.585458:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328448:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048051.585494:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328512:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048051.585542:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048051.585549:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513328384:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048051.586976:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513328384:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048051.587050:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048051.587054:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048051.587058:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328512:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048051.587318:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328448:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048052.225312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048052.481236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048053.249364:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048053.505287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048054.273283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048054.529237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048055.297323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048055.553235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048056.321333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048056.577251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048056.705351:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048056.705358:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048056.705362:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048056.705373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1692 00000100:00080000:2.0:1781048056.705377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 11s 00000100:00100000:3.0:1781048056.705447:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048056.705558:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048056.705565:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513328640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048056.706618:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048056.706635:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048056.706668:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513328704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048056.706695:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048056.706699:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048056.706744:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328704:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048056.706851:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513328640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048056.706881:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048056.707605:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048056.707610:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048056.707615:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328704:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048057.345305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048057.601247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048058.369287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048058.625234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048059.393301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048059.649254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048060.417298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048060.673230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048061.441333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048061.697241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048061.825263:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048061.825270:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048061.825274:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048061.825289:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1697 00000100:00080000:2.0:1781048061.825293:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00080000:2.0:1781048061.825301:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048061.825301:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328768:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781048061.825313:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048061.825330:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000521535b2 x1867562513328896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048061.825350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048061.825353:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048061.825383:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328896:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048061.825388:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048061.826806:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513328768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048061.827229:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048061.827233:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048061.827237:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513328896:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048061.827248:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513328832:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781048062.465311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048062.721291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048063.489301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048063.745281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048064.513302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048064.769182:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048065.537255:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048065.793282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048066.561325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048066.817251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048066.945300:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048066.945308:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048066.945312:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048066.945324:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513328960:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048066.945324:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1702 00000100:00080000:1.0:1781048066.945329:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048066.945336:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048066.945346:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048066.945369:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ce8e0ca x1867562513329088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048066.945387:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048066.945389:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048066.945412:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048066.945421:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329088:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048066.945423:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513329024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048066.946540:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513329024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048066.946553:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513328960:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048066.946699:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048066.946703:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048066.946706:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329088:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048067.585275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048067.841357:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048068.609316:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048068.865243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048069.633352:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048069.889220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048070.657732:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048070.913240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048071.682367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048071.937284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048072.065337:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048072.065345:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048072.065348:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048072.065358:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1707 00000100:00080000:2.0:1781048072.065362:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781048072.065364:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513329152:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048072.065371:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048072.065382:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048072.065397:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018a334bb x1867562513329280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048072.065418:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048072.065422:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048072.065442:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048072.065451:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513329216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048072.065467:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329280:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048072.066679:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513329216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048072.066680:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048072.066687:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048072.066694:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329280:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048072.066699:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513329152:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781048072.705295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048072.961229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048073.729318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048073.985253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048074.753326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048075.009239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048075.777295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048076.033212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048076.801331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048077.057251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048077.185323:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048077.185331:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048077.185335:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048077.185344:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1713 00000100:00080000:1.0:1781048077.185348:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048077.185357:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781048077.185358:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513329344:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781048077.185367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048077.185382:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513329472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048077.185411:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048077.185415:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048077.185437:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513329408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048077.185458:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329472:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048077.186886:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513329344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048077.186890:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513329408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048077.187119:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048077.187156:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048077.187161:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329472:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048077.825318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048078.081226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048078.849300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048079.105285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048079.873308:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048080.129273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048080.897283:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048081.153223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048081.921299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048082.177259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048082.305283:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048082.305290:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048082.305293:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048082.305303:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1718 00000100:00080000:1.0:1781048082.305306:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048082.305311:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781048082.305313:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513329536:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781048082.305320:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048082.305332:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513329664/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048082.305353:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048082.305356:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048082.305377:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513329600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048082.305393:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329664:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048082.306520:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513329536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048082.306761:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513329600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048082.311745:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048082.311752:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048082.311758:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329664:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048082.945288:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048083.201217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048083.969281:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048084.225222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048084.993322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048085.249253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048086.017355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048086.273251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048087.041264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048087.297224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048087.425320:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048087.425326:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048087.425328:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048087.425338:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1723 00000100:00080000:3.0:1781048087.425340:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048087.425345:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048087.425350:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513329792:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:3.0:1781048087.425355:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048087.425397:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513329856/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048087.425415:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048087.425418:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048087.425437:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048087.425443:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513329728:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048087.425490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329856:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048087.426470:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513329728:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048087.426483:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513329792:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048087.426518:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048087.426522:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048087.426527:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513329856:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048088.065316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048088.321245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048089.089310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048089.345239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048090.113354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048090.369252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048091.137250:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048091.393184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048092.161307:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048092.417229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048092.545260:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048092.545267:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048092.545271:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781048092.545294:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513329920:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048092.545296:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1728 00000100:00080000:0.0:1781048092.545301:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048092.545309:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048092.545320:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048092.545338:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513330048/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048092.545357:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048092.545361:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048092.545391:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513329984:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048092.545396:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330048:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048092.546629:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513329920:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048092.546643:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513329984:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048092.546656:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048092.546661:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048092.546665:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330048:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048093.185228:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048093.441306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048094.209323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048094.465225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048095.233317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048095.489217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048096.257334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048096.513260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048097.281332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048097.537239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048097.665321:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048097.665327:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048097.665331:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048097.665341:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1733 00000100:00100000:1.0:1781048097.665355:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513330112:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048097.665358:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048097.665366:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048097.665375:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048097.665391:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513330240/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048097.665409:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048097.665413:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048097.665438:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513330176:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048097.665455:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330240:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048097.666771:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513330112:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048097.666780:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513330176:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:02020000:2.0:1781048097.666805:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:2.0:1781048097.670168:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048097.670196:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048097.670212:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330240:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048098.305331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048098.561246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048099.329326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048099.585266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048100.353324:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048100.609287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048101.377381:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048101.633301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048102.401339:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048102.657288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048102.785340:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048102.785348:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048102.785352:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048102.785365:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1738 00000100:00100000:0.0:1781048102.785367:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513330304:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048102.785371:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048102.785391:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048102.785403:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048102.785421:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513330432/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048102.785444:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048102.785448:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048102.785483:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330432:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048102.785487:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513330368:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048102.786978:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513330368:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048102.786989:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513330304:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048102.787013:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048102.787019:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048102.787024:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330432:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048103.425360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048103.681325:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048104.449362:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048104.705334:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048105.473370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048105.729315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048106.497321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048106.753299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048107.521357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048107.777275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048107.905371:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048107.905379:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048107.905382:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048107.905394:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1743 00000100:00080000:2.0:1781048107.905397:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048107.905403:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048107.905414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048107.905425:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513330496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048107.905432:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513330624/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048107.905464:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048107.905467:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048107.905489:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513330560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048107.905511:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330624:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048107.908850:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513330496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048107.908893:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513330560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048107.908926:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048107.908936:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048107.908941:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330624:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048108.545282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048108.801234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048109.569366:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048109.825298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048110.593414:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048110.849282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048111.617377:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048111.873284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048112.641316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048112.897276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048113.025301:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048113.025327:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048113.025333:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781048113.025340:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513330688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048113.025342:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1748 00000100:00080000:2.0:1781048113.025346:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048113.025353:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048113.025362:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048113.025378:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000523c4981 x1867562513330816/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048113.025393:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048113.025396:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048113.025421:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330816:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048113.025436:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513330752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048113.026722:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513330688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048113.026772:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513330752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048113.026798:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048113.026802:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048113.026806:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513330816:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048113.665316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048113.921339:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048114.689417:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048114.945373:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048115.713460:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048115.969241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048116.737311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048116.993187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048117.761303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048118.017297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048118.145262:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048118.145268:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048118.145272:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048118.145281:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1754 00000100:00080000:2.0:1781048118.145283:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781048118.145288:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513330880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048118.145290:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048118.145302:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048118.145314:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513331008/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048118.145361:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048118.145364:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048118.145386:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513330944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048118.145402:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048118.146367:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513330880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048118.146460:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513330944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048118.146488:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048118.146493:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048118.146498:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331008:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048118.785314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048119.041228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048119.809275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048120.065233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048120.833283:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048121.089248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048121.857330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048122.113247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048122.881291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048123.137233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048123.265317:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048123.265324:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048123.265328:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048123.265338:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1759 00000100:00080000:0.0:1781048123.265342:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048123.265348:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048123.265358:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048123.265359:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331072:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048123.265377:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513331200/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048123.265396:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048123.265398:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048123.265432:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331136:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048123.265436:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331200:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048123.267777:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331072:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048123.267863:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331136:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048123.267883:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048123.267889:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048123.267894:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331200:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048123.905627:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048124.161227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048124.929311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048125.185234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048125.953293:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048126.209246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048126.977315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048127.233254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048128.001297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048128.257248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048128.385329:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048128.385336:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048128.385339:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048128.385349:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1764 00000100:00100000:0.0:1781048128.385351:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331264:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048128.385353:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048128.385362:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048128.385372:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048128.385390:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513331392/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048128.385412:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048128.385416:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048128.385453:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331392:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048128.385457:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331328:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048128.386766:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331264:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048128.386778:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331328:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048128.386797:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048128.386808:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048128.386832:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331392:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048129.025330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048129.281265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048130.049264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048130.305281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048131.073335:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048131.329254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048132.097338:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048132.353273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048133.121359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048133.377283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048133.505340:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048133.505350:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048133.505354:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781048133.505360:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331456:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048133.505365:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1769 00000100:00080000:0.0:1781048133.505386:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048133.505394:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048133.505407:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048133.505425:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513331584/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048133.505444:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048133.505448:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048133.505473:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331520:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048133.505483:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331584:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048133.508609:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331456:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048133.508609:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781048133.508617:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331520:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048133.508620:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048133.508628:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331584:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048134.145326:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048134.401243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048135.169328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048135.425287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048136.193368:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048136.449280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048137.217324:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048137.473269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048138.241358:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048138.497248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048138.625375:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048138.625382:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048138.625386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048138.625396:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1774 00000100:00080000:1.0:1781048138.625399:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048138.625408:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781048138.625409:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331648:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781048138.625418:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048138.625433:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513331776/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048138.625453:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048138.625456:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048138.625507:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048138.625514:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331776:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048138.626957:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331648:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048138.626974:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048138.627002:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048138.627012:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048138.627017:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331776:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048139.265372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048139.521283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048140.289344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048140.545253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048141.313264:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048141.569250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048142.337326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048142.593267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048143.361367:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048143.617224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048143.745361:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048143.745397:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048143.745401:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048143.745411:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048143.745411:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1779 00000100:00080000:1.0:1781048143.745416:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048143.745424:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048143.745438:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048143.745455:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b98b0b8e x1867562513331968/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048143.745501:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048143.745505:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048143.745530:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048143.745545:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331968:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048143.746841:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513331840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048143.746857:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513331904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048143.746881:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048143.746892:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048143.746896:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513331968:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048144.385429:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048144.641275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048145.409301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048145.665265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048146.433351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048146.689274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048147.457323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048147.713264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048148.481343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048148.737275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048148.865365:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048148.865374:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048148.865378:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048148.865400:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513332032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048148.865400:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1784 00000100:00080000:1.0:1781048148.865406:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048148.865415:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048148.865431:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048148.865450:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513332160/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048148.865476:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048148.865479:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048148.865505:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513332096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048148.865527:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332160:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048148.866983:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513332032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048148.866984:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513332096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048148.867011:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048148.867016:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048148.867021:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332160:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048149.505262:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048149.761207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048150.529328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048150.785270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048151.553288:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048151.809198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048152.577360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048152.833271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048153.601361:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048153.857268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048153.985310:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048153.985319:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048153.985322:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048153.985355:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513332224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048153.985356:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1789 00000100:00080000:1.0:1781048153.985360:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048153.985366:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048153.985375:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048153.985393:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513332352/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048153.985420:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048153.985424:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048153.985447:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513332288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048153.985509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332352:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048153.986698:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513332224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048153.986755:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513332288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048153.986977:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048153.986987:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048153.986992:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332352:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048154.625296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048154.881251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048155.649323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048155.905253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048156.673359:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048156.929283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048157.697317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048157.953268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048158.721310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048158.977260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048159.105336:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048159.105345:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048159.105348:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048159.105360:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1795 00000100:00080000:3.0:1781048159.105365:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048159.105372:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048159.105382:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048159.105399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513332416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048159.105400:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513332544/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048159.105449:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048159.105453:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048159.105496:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513332480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048159.105649:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332544:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048159.107475:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513332416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048159.107565:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513332480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048159.107598:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048159.107602:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048159.107606:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332544:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048159.745292:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048160.001227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048160.769312:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048161.025240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048161.793332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048162.049242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048162.817347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048163.073238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048163.841325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048164.097271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048164.225322:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048164.225338:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048164.225342:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048164.225357:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1800 00000100:00100000:2.0:1781048164.225361:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513332672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048164.225364:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048164.225374:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048164.225387:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048164.225429:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513332736/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048164.225450:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048164.225454:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048164.225466:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048164.225475:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513332608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048164.225490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332736:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048164.226774:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513332608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048164.226792:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781048164.226793:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513332672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048164.226799:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048164.226805:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332736:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048164.865328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048165.121212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048165.889326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048166.145265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048166.913306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048167.169259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048167.937313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048168.193259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048168.961287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048169.217225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048169.345277:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048169.345282:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048169.345285:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048169.345292:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1805 00000100:00080000:3.0:1781048169.345295:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048169.345300:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048169.345308:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048169.345320:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513332928/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781048169.345328:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513332864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048169.345335:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048169.345339:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048169.345378:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513332800:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048169.345391:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332928:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048169.346654:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513332864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048169.346676:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048169.346683:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048169.346688:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513332800:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048169.346690:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513332928:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048169.985278:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048170.241223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048171.009341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048171.265237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048172.033330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048172.289252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048173.057399:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048173.313256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048174.081316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048174.337278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048174.465370:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048174.465377:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048174.465381:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048174.465391:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1810 00000100:00100000:0.0:1781048174.465392:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513332992:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048174.465395:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048174.465402:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048174.465412:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048174.465427:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513333120/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048174.465445:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048174.465448:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048174.465468:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513333056:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048174.465492:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333120:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048174.466947:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513332992:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048174.466959:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513333056:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048174.466978:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048174.466983:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048174.466988:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333120:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048175.105376:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048175.361274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048176.129370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048176.385249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048177.153341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048177.409276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048178.177318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048178.433285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048179.201342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048179.457258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048179.585347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048179.585357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048179.585361:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048179.585372:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1815 00000100:00080000:3.0:1781048179.585375:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048179.585382:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048179.585389:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333184:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048179.585393:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048179.585408:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513333312/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048179.585426:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048179.585429:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048179.585465:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513333248:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048179.585478:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333312:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048179.586789:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048179.587400:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048179.587405:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048179.587409:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333312:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048179.587654:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513333248:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781048180.225244:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048180.481271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048181.249304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048181.505247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048182.273313:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048182.529223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048183.297289:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048183.553248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048184.321294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048184.577244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048184.705347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048184.705355:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048184.705360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048184.705374:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1820 00000100:00080000:3.0:1781048184.705378:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048184.705404:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048184.705413:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333376:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048184.705419:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048184.705442:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513333504/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048184.705464:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048184.705468:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048184.705510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333504:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048184.705512:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513333440:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048184.710405:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048184.710420:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513333440:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048184.710463:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048184.710472:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048184.710479:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333504:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048185.345301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048185.601232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048186.369276:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048186.625245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048187.393334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048187.649259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048188.417321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048188.673256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048189.441322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048189.697255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048189.825266:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048189.825273:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048189.825277:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048189.825287:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1825 00000100:00080000:3.0:1781048189.825291:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1781048189.825297:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048189.825298:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048189.825311:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048189.825324:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513333696/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048189.825337:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048189.825340:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048189.825353:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513333632:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048189.825371:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333696:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048189.826638:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048189.826686:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513333632:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048189.826719:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048189.826723:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048189.826728:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333696:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048190.465313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048190.721268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048191.489329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048191.745279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048192.513320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048192.769241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048193.537271:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048193.793249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048194.561331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048194.817199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048194.945349:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048194.945356:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048194.945359:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048194.945371:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513333760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048194.945371:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1830 00000100:00080000:1.0:1781048194.945377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781048194.945386:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048194.945394:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048194.945409:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084471047 x1867562513333888/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048194.945427:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048194.945429:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048194.945454:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513333824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048194.945462:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333888:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048194.946648:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513333760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048194.946732:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781048194.946733:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513333824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048194.946737:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048194.946747:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513333888:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048195.585292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048195.841214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048196.609334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048196.865252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048197.633274:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048197.889240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048198.657303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048198.913188:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048199.681288:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048199.937236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048200.065341:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048200.065348:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048200.065351:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048200.065362:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1835 00000100:00080000:2.0:1781048200.065366:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048200.065371:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048200.065386:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333952:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:2.0:1781048200.065409:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048200.065425:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513334080/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048200.065442:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048200.065444:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048200.065464:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513334016:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048200.065487:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334080:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048200.066688:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513334016:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048200.066732:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513333952:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048200.066758:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048200.066762:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048200.066766:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334080:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048200.706258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048200.961248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048201.729265:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048201.985201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048202.753317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048203.009254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048203.777384:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048204.033226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048204.801333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048205.057221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048205.185339:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048205.185349:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048205.185353:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048205.185366:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1841 00000100:00080000:2.0:1781048205.185370:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781048205.185374:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513334144:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048205.185380:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048205.185393:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048205.185410:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513334272/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048205.185431:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048205.185452:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048205.185472:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334272:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048205.185474:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048205.185482:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513334208:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048205.186846:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513334208:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048205.186853:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048205.186858:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048205.186865:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334272:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048205.186866:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513334144:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048205.825328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048206.081231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048206.849347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048207.105272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048207.873337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048208.129212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048208.897331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048209.153257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048209.921331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048210.177282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048210.305351:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048210.305360:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048210.305363:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048210.305377:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1846 00000100:00080000:0.0:1781048210.305381:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048210.305389:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048210.305403:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513334336:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048210.305407:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048210.305428:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513334464/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048210.305449:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048210.305473:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048210.305480:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334464:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048210.305522:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513334400:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048210.306831:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513334336:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048210.306907:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513334400:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048210.306929:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048210.306935:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048210.306941:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334464:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048210.945349:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048211.201274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048211.969341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048212.225253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048212.993337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048213.249260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048214.017327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048214.273279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048215.041301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048215.297213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048215.425345:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048215.425354:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048215.425359:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048215.425371:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1851 00000100:00080000:1.0:1781048215.425375:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781048215.425379:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513334528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048215.425385:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048215.425396:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048215.425413:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001d23a43b x1867562513334656/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048215.425435:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048215.425438:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048215.425457:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513334592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048215.425490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334656:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048215.426960:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513334528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048215.427029:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048215.427036:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048215.427039:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513334592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048215.427042:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334656:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048216.065338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048216.321244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048217.089253:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048217.345226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048218.113294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048218.369251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048219.137332:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048219.393244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048220.161321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048220.417267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048220.545333:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048220.545340:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048220.545344:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048220.545354:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1856 00000100:00080000:1.0:1781048220.545357:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781048220.545361:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513334720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048220.545366:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048220.545377:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048220.545391:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513334848/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048220.545423:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048220.545426:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048220.545447:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513334784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048220.545471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334848:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048220.546956:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513334720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048220.546977:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513334784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048220.546995:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048220.547001:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048220.547006:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513334848:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048221.185357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048221.441217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048222.209312:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048222.465309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048223.233334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048223.489235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048224.257330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048224.513262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048225.281334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048225.537197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048225.665352:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048225.665365:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048225.665370:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048225.665384:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513334912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048225.665384:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1861 00000100:00080000:3.0:1781048225.665390:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048225.665398:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048225.665410:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048225.665447:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513335040/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048225.665475:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048225.665479:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048225.665481:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048225.665487:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513334976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048225.665518:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335040:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781048225.667088:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048225.667096:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048225.667097:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513334976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048225.667101:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335040:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048225.667107:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513334912:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781048226.305318:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048226.561277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048227.329338:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048227.585270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048228.353347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048228.609266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048229.377308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048229.633258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048230.401442:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048230.657287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048230.785310:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048230.785317:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048230.785320:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048230.785331:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1866 00000100:00080000:3.0:1781048230.785335:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048230.785341:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048230.785345:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335168:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048230.785351:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048230.785370:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513335232/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048230.785392:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048230.785396:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048230.785446:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335232:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048230.785447:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048230.785459:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513335104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048230.786808:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513335104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048230.786828:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048230.786855:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048230.786860:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048230.786866:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335232:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048231.425339:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048231.681279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048232.449269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048232.705244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048233.473334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048233.729268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048234.497271:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048234.753248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048235.521337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048235.777304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048235.905327:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048235.905337:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048235.905341:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048235.905351:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1871 00000100:00100000:3.0:1781048235.905352:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513335296:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048235.905355:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048235.905365:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048235.905378:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048235.905395:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb943f69 x1867562513335424/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048235.905412:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048235.905415:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048235.905435:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335360:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048235.905462:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335424:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048235.906646:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513335296:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048235.906651:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335360:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048235.906674:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048235.906679:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048235.906683:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335424:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048236.545283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048236.801242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048237.569320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048237.825186:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048238.593261:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048238.849214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048239.617344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048239.873262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048240.641351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048240.897252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048241.025315:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048241.025322:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048241.025325:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048241.025337:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1876 00000100:00080000:2.0:1781048241.025341:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048241.025348:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048241.025360:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048241.025364:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513335488:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048241.025380:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e5c8472 x1867562513335616/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048241.025408:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048241.025412:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048241.025446:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335616:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048241.025456:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335552:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048241.026755:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335552:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048241.026817:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513335488:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048241.026836:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048241.026840:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048241.026845:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335616:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048241.665325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048241.921262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048242.689363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048242.945232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048243.713342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048243.969277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048244.737318:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048244.993207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048245.761305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048246.017280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048246.145311:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048246.145320:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048246.145325:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048246.145337:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1882 00000100:00080000:0.0:1781048246.145342:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048246.145349:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048246.145354:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513335680:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048246.145362:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048246.145379:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018617087 x1867562513335808/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048246.145401:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048246.145423:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048246.145467:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513335744:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048246.145515:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335808:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048246.147105:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513335680:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048246.147119:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513335744:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048246.147167:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048246.147173:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048246.147178:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513335808:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048246.785333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048247.041206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048247.809299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048248.065241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048248.833276:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048249.089254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048249.857369:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048250.113250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048250.881323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048251.137288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048251.265398:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048251.265405:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048251.265409:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781048251.265433:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513335872:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048251.265437:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1887 00000100:00080000:2.0:1781048251.265443:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048251.265450:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048251.265464:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048251.265510:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513336000/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048251.265532:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048251.265535:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048251.265568:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335936:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048251.265585:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336000:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048251.267161:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513335936:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048251.267168:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513335872:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1781048251.267177:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781048251.271514:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048251.271524:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048251.271533:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336000:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048251.905301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048252.161272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048252.929326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048253.185250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048253.953327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048254.209267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048254.977366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048255.233270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048256.001324:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048256.257295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048256.385371:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048256.385391:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048256.385395:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048256.385407:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1892 00000100:00080000:0.0:1781048256.385410:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048256.385415:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048256.385420:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513336064:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048256.385425:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048256.385439:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513336192/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048256.385457:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048256.385476:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048256.385498:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336192:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048256.385499:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513336128:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048256.387032:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513336128:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048256.387032:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513336064:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048256.387032:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048256.387040:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048256.387046:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336192:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048257.025339:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048257.281263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048258.049327:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048258.305239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048259.073278:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048259.329278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048260.097322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048260.353239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048261.121345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048261.377250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048261.505324:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048261.505335:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048261.505339:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048261.505351:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1897 00000100:00100000:2.0:1781048261.505356:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513336320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048261.505356:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048261.505367:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048261.505380:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048261.505401:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513336384/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048261.505422:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048261.505425:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048261.505446:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336384:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048261.505463:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513336256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048261.506826:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513336320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048261.506843:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513336256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048261.506876:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048261.506881:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048261.506887:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336384:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048262.145358:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048262.401196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048263.169355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048263.425295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048264.193311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048264.449231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048265.217290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048265.473244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048266.241323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048266.497257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048266.625295:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048266.625300:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048266.625303:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048266.625310:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1902 00000100:00080000:3.0:1781048266.625312:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048266.625317:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048266.625325:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048266.625333:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513336512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048266.625338:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000048c6b8a x1867562513336576/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048266.625353:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048266.625356:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048266.625395:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336576:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048266.625402:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513336448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048266.626789:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513336512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048266.626789:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513336448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048266.626799:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048266.626803:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048266.626808:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336576:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048267.265297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048267.521260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048268.289291:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048268.545230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048269.313273:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048269.569231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048270.337303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048270.593241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048271.361262:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048271.617233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048271.745349:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048271.745358:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048271.745363:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048271.745377:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1907 00000100:00080000:0.0:1781048271.745385:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048271.745393:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048271.745415:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513336640:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048271.745424:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048271.745446:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b24224a5 x1867562513336768/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048271.745469:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048271.745472:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048271.745497:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513336704:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048271.745527:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336768:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048271.746853:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513336640:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048271.746855:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513336704:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048271.746878:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048271.746893:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048271.746898:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336768:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048272.385240:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048272.641224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048273.409240:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048273.665227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048274.433276:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048274.689226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048275.457257:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048275.713229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048276.481292:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048276.737245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048276.865286:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048276.865292:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048276.865295:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048276.865303:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1912 00000100:00080000:0.0:1781048276.865309:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048276.865314:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048276.865324:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048276.865340:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513336960/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048276.865361:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048276.865365:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048276.865379:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513336832:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048276.865394:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513336896:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048276.865408:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336960:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048276.866653:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513336896:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048276.866666:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513336832:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048276.866731:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048276.866736:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048276.866740:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513336960:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048277.506145:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048277.761190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048278.529277:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048278.785229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048279.553323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048279.809181:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048280.577306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048280.833238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048281.601272:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048281.857185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048281.985309:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048281.985317:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048281.985320:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048281.985331:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1917 00000100:00080000:3.0:1781048281.985334:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048281.985353:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513337088:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048281.985356:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048281.985367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048281.985397:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513337152/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048281.985416:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048281.985419:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048281.985436:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048281.985444:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513337024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048281.985446:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337152:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048281.986698:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513337088:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048281.986723:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513337024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048281.986756:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048281.986761:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048281.986765:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337152:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048282.625299:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048282.881243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048283.649292:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048283.905187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048284.673275:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048284.929236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048285.697326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048285.953267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048286.721333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048286.977285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048287.105350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048287.105359:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048287.105363:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048287.105374:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1923 00000100:00100000:1.0:1781048287.105374:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048287.105380:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048287.105389:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048287.105400:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048287.105415:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513337344/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048287.105434:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048287.105437:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048287.105458:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513337280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048287.105482:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337344:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048287.106796:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048287.106818:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513337280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048287.106858:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048287.106863:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048287.106868:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337344:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048287.745312:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048288.001259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048288.769309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048289.025243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048289.793318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048290.049199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048290.817314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048291.073250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048291.841314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048292.097259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048292.225358:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048292.225368:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048292.225373:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048292.225374:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048292.225384:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1928 00000100:00080000:1.0:1781048292.225387:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048292.225412:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048292.225425:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048292.225443:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f819707f x1867562513337536/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048292.225466:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048292.225470:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048292.225473:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781048292.225480:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513337472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048292.225516:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337536:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048292.226844:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513337472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048292.226867:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048292.226868:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048292.226882:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048292.226889:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337536:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048292.865275:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048293.121232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048293.889296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048294.145231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048294.913307:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048295.169217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048295.937293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048296.193221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048296.961237:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048297.217221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048297.345292:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048297.345299:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048297.345303:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048297.345313:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1933 00000100:00080000:1.0:1781048297.345317:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048297.345324:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781048297.345327:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337600:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781048297.345335:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048297.345351:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513337728/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048297.345372:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048297.345376:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048297.345395:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513337664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048297.345413:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337728:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048297.346572:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048297.346611:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513337664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048297.346638:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048297.346643:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048297.346648:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337728:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048297.985289:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048298.241226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048299.009314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048299.265219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048300.033307:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048300.289222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048301.057278:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048301.313227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048302.081337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048302.337259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048302.465307:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048302.465316:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048302.465320:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048302.465331:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513337856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048302.465332:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1938 00000100:00080000:3.0:1781048302.465336:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048302.465344:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048302.465356:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048302.465373:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513337920/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048302.465397:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048302.465401:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048302.465440:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781048302.465447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337920:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048302.465452:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513337792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048302.466553:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513337792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048302.466583:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781048302.466591:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513337856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048302.466593:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048302.466602:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513337920:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048303.105325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048303.361248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048304.129318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048304.385261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048305.153309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048305.409247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048306.177295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048306.433289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048307.201350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048307.457260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048307.585325:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048307.585332:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048307.585335:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048307.585345:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1943 00000100:00080000:0.0:1781048307.585348:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781048307.585353:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337984:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048307.585355:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048307.585382:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048307.585398:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003cae67cc x1867562513338112/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048307.585414:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048307.585417:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048307.585452:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048307.585458:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338112:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048307.585464:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513338048:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048307.586599:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513338048:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048307.586612:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513337984:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048307.586614:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048307.586619:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048307.586624:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338112:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048308.225317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048308.481293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048309.249362:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048309.505273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048310.273270:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048310.529297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048311.297377:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048311.553269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048312.321248:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048312.577271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048312.705370:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048312.705392:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048312.705397:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048312.705412:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1948 00000100:00080000:0.0:1781048312.705416:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048312.705425:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048312.705429:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513338176:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048312.705438:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048312.705461:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513338304/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048312.705486:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048312.705490:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048312.705528:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338304:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048312.705535:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513338240:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048312.706747:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513338176:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048312.706836:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513338240:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048312.706853:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048312.706874:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048312.706879:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338304:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048313.345386:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048313.601297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048314.369275:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048314.625230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048315.393394:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048315.649288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048316.417365:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048316.673204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048317.441353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048317.697287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048317.825327:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048317.825337:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048317.825342:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048317.825356:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1953 00000100:00080000:3.0:1781048317.825358:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048317.825363:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513338432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048317.825365:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048317.825469:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048317.825484:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048317.825498:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513338496/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781048317.825500:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513338368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048317.825529:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048317.825533:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048317.825574:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338496:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048317.826735:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513338368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048317.826748:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513338432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048317.826823:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048317.826827:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048317.826831:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338496:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048318.465319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048318.721281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048319.489353:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048319.745266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048320.513333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048320.769270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048321.537367:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048321.793280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048322.561394:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048322.817272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048322.945362:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048322.945369:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048322.945373:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048322.945385:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1958 00000100:00080000:3.0:1781048322.945390:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048322.945399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513338560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048322.945399:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048322.945414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048322.945434:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513338688/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048322.945452:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048322.945455:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048322.945483:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338688:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048322.945488:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513338624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048322.946767:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513338560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048322.946768:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513338624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048322.946790:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048322.946794:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048322.946799:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338688:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048323.585382:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048323.841277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048324.609352:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048324.865204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048325.633403:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048325.889219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048326.657387:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048326.913284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048327.681336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048327.937255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048328.065275:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048328.065285:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048328.065289:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048328.065299:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513338752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048328.065299:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1963 00000100:00080000:3.0:1781048328.065321:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048328.065329:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048328.065343:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048328.065362:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513338880/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048328.065381:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048328.065385:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048328.065407:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338880:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048328.065411:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513338816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048328.066701:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513338752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048328.066715:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513338816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048328.066735:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048328.066739:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048328.066743:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513338880:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048328.705337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048328.961266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048329.729325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048329.985256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048330.753335:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048331.009215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048331.777274:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048332.033247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048332.801346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048333.057275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048333.185350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048333.185357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048333.185361:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048333.185377:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1969 00000100:00100000:2.0:1781048333.185379:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513338944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048333.185381:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048333.185390:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048333.185400:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048333.185415:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9891000 x1867562513339072/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048333.185429:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048333.185432:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048333.185455:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048333.185469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048333.186681:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513338944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048333.186727:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048333.186742:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048333.186747:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048333.186753:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339072:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048333.825353:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048334.081231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048334.849356:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048335.105268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048335.873359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048336.129193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048336.897290:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048337.153260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048337.921332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048338.177216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048338.305299:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048338.305308:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048338.305312:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048338.305325:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1974 00000100:00100000:2.0:1781048338.305326:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048338.305330:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048338.305339:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048338.305350:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048338.305370:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513339264/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048338.305396:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048338.305400:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048338.305410:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048338.305416:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513339200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048338.305443:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339264:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048338.306941:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048338.306978:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513339200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048338.307021:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048338.307026:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048338.307031:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339264:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048338.945309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048339.201257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048339.969341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048340.225242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048340.993335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048341.249260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048342.017337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048342.273288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048343.041316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048343.297240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048343.425350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048343.425357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048343.425360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048343.425373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1979 00000100:00100000:2.0:1781048343.425374:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339328:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048343.425381:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048343.425388:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048343.425398:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048343.425414:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513339456/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048343.425433:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048343.425442:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048343.425466:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048343.425481:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339456:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048343.426855:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048343.426863:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339328:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048343.426869:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048343.426876:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048343.426883:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339456:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048344.065363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048344.321266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048345.089319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048345.345263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048346.113338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048346.369270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048347.137396:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048347.393384:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048348.161410:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048348.417264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048348.545360:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048348.545370:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048348.545375:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048348.545388:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1984 00000100:00100000:2.0:1781048348.545389:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048348.545393:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048348.545401:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048348.545415:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048348.545434:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513339648/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048348.545458:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048348.545462:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048348.545490:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048348.545501:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339648:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048348.546859:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048348.546875:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048348.546914:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048348.546919:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048348.546923:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339648:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048349.185269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048349.441236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048350.209351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048350.465292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048351.233301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048351.489270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048352.257391:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048352.513277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048353.281334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048353.537220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048353.665311:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048353.665318:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048353.665321:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048353.665332:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1989 00000100:00080000:3.0:1781048353.665335:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048353.665344:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048353.665359:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048353.665371:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048353.665389:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513339840/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048353.665412:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048353.665416:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048353.665440:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048353.665462:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339840:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048353.666822:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513339776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048353.666842:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513339712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048353.666881:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048353.666886:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048353.666891:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513339840:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048354.305357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048354.561282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048355.329272:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048355.585253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048356.353332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048356.609267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048357.377349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048357.633263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048358.401320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048358.657251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048358.785347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048358.785356:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048358.785360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048358.785370:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513339904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048358.785375:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1994 00000100:00080000:1.0:1781048358.785380:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048358.785387:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048358.785401:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048358.785421:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513340032/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048358.785467:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048358.785475:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048358.785489:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781048358.785497:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513339968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048358.785507:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340032:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048358.786692:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513339968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048358.786704:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513339904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048358.786764:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048358.786769:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048358.786774:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340032:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048359.425330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048359.681259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048360.449351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048360.705267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048361.473346:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048361.729247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048362.497339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048362.753281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048363.521336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048363.777269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048363.905333:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048363.905343:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048363.905347:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048363.905360:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 1999 00000100:00080000:3.0:1781048363.905364:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048363.905368:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048363.905373:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048363.905385:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048363.905399:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513340224/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048363.905418:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048363.905421:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048363.905440:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048363.905448:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340224:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048363.906826:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048363.906840:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048363.906845:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048363.906851:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048363.906856:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340224:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048364.545337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048364.801283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048365.569359:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048365.825249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048366.593276:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048366.849241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048367.617332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048367.873197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048368.641266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048368.897235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048369.025371:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048369.025381:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048369.025414:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048369.025421:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048369.025426:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2004 00000100:00080000:3.0:1781048369.025432:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048369.025440:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048369.025455:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048369.025476:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513340416/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048369.025500:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048369.025503:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048369.025535:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048369.025539:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781048369.027187:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781048369.027189:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048369.027194:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048369.027201:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048369.027207:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340288:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781048369.665337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048369.921254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048370.689337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048370.945207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048371.713363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048371.969251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048372.737353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048372.993262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048373.761345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048374.017289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048374.145268:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048374.145274:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048374.145277:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048374.145290:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2010 00000100:00080000:3.0:1781048374.145305:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048374.145314:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048374.145322:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340480:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048374.145326:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048374.145345:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513340608/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048374.145383:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048374.145386:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048374.145420:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340608:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048374.145426:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048374.146772:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781048374.146773:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048374.146777:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048374.146783:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340608:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048374.146790:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340544:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781048374.785374:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048375.041206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048375.809327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048376.065299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048376.833351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048377.089266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048377.857349:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048378.113213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048378.881260:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048379.137276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048379.265354:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048379.265364:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048379.265368:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048379.265381:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2015 00000100:00080000:3.0:1781048379.265385:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048379.265389:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048379.265393:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048379.265405:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048379.265421:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513340800/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048379.265440:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048379.265443:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048379.265464:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340736:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048379.265480:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340800:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048379.266823:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340736:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048379.266824:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048379.266829:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048379.266833:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048379.266838:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340800:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048379.905349:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048380.161271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048380.929299:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048381.185256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048381.953295:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048382.209244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048382.977377:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048383.233239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048384.001326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048384.257272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048384.385316:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048384.385325:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048384.385329:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048384.385347:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048384.385355:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2020 00000100:00080000:3.0:1781048384.385359:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048384.385366:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048384.385379:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048384.385397:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513340992/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048384.385421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048384.385424:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048384.385446:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048384.385475:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340992:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048384.386782:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513340864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048384.386782:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513340928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048384.386798:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048384.386803:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048384.386809:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513340992:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781048385.026169:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048385.281257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048386.049295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048386.305242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048387.073336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048387.329273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048388.097273:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048388.353272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048389.121291:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048389.377276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048389.505337:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048389.505347:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048389.505351:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048389.505362:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2025 00000100:00080000:3.0:1781048389.505366:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048389.505371:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048389.505381:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048389.505397:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513341184/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048389.505419:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048389.505429:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048389.505462:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513341120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048389.505469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341184:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048389.505598:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513341056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048389.507208:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513341056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048389.507265:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048389.507271:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048389.507276:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341184:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048389.508957:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513341120:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781048390.145346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048390.401241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048391.169342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048391.425264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048392.193319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048392.449182:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048393.217290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048393.473253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048394.241299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048394.497230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048394.625369:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048394.625376:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048394.625380:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048394.625390:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2030 00000100:00080000:0.0:1781048394.625393:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048394.625399:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048394.625403:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513341248:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048394.625410:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048394.625427:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513341376/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048394.625449:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048394.625453:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048394.625474:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513341312:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048394.625494:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341376:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048394.627005:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513341248:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048394.627047:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048394.627052:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048394.627057:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341376:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048394.627100:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513341312:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1781048395.265295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048395.521333:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048396.289275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048396.545229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048397.313318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048397.569229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048398.337299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048398.593221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048399.361273:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048399.617264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048399.745362:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048399.745371:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048399.745375:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048399.745390:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2035 00000100:00100000:1.0:1781048399.745391:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513341440:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048399.745395:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048399.745404:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048399.745417:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048399.745434:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513341568/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048399.745477:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048399.745481:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048399.745507:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513341504:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048399.745520:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341568:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048399.746636:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513341440:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048399.746655:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513341504:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048399.746774:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048399.746778:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048399.746781:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341568:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048400.385362:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048400.641262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048401.409305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048401.665280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048402.433315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048402.689269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048403.457308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048403.713239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048404.481331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048404.737247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048404.865352:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048404.865359:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048404.865362:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048404.865374:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2040 00000100:00080000:0.0:1781048404.865377:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048404.865383:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048404.865393:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513341632:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048404.865394:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048404.865407:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513341760/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048404.865424:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048404.865427:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048404.865440:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513341696:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048404.865469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341760:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048404.866911:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513341696:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048404.866930:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513341632:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:02020000:3.0:1781048404.867019:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:3.0:1781048404.869950:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048404.869961:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048404.869972:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341760:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048405.505287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048405.761210:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048406.529350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048406.785258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048407.553348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048407.809209:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048408.577297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048408.833273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048409.602828:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048409.857229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048409.985325:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048409.985336:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048409.985341:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048409.985353:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2045 00000100:00080000:3.0:1781048409.985356:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048409.985370:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513341824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048409.985379:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048409.985391:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048409.985406:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000288f7157 x1867562513341952/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048409.985421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048409.985424:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048409.985438:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513341888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048409.985449:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341952:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048409.986669:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513341824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048409.986750:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513341888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048409.986788:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048409.986793:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048409.986798:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513341952:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048410.625325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048410.881277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048411.649339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048411.905187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048412.673264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048412.929254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048413.697331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048413.953258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048414.721328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048414.977272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048415.105280:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048415.105286:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048415.105290:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048415.105301:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2051 00000100:00080000:0.0:1781048415.105304:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781048415.105309:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513342016:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048415.105311:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048415.105323:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048415.105336:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513342144/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048415.105352:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048415.105355:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048415.105376:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513342080:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048415.105392:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342144:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048415.106586:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513342016:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048415.106600:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513342080:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048415.106627:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048415.106632:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048415.106638:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342144:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:0.0:1781048415.745307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048416.001220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048416.769307:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048417.025273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048417.793346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048418.049272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048418.817311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048419.073239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048419.841331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048420.097230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048420.225322:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048420.225332:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048420.225336:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048420.225348:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2056 00000100:00080000:2.0:1781048420.225353:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781048420.225356:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513342208:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048420.225362:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048420.225375:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048420.225392:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084e7220c x1867562513342336/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048420.225437:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048420.225440:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048420.225470:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342336:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048420.225489:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048420.225497:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513342272:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048420.226497:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513342208:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048420.226507:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048420.226517:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048420.226522:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342336:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048420.226531:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513342272:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:1.0:1781048420.865320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048421.121291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048421.889307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048422.145227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048422.913279:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048423.169225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048423.937323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048424.193277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781048424.961301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048425.217266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048425.345350:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048425.345357:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048425.345361:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048425.345372:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2061 00000100:00080000:0.0:1781048425.345375:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048425.345382:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048425.345406:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048425.345424:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513342528/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781048425.345439:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513342400:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048425.345451:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048425.345456:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048425.345487:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513342464:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048425.345494:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342528:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048425.346636:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513342400:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048425.346771:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048425.346776:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048425.346781:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342528:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048425.346790:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513342464:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048425.985320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048426.241269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048427.009280:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048427.265227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048428.033363:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048428.289284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048429.057367:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048429.313191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048430.081366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048430.337222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048430.465344:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048430.465352:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048430.465356:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048430.465366:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2066 00000100:00080000:2.0:1781048430.465371:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048430.465376:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048430.465386:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048430.465399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513342592:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048430.465404:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513342720/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048430.465434:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048430.465439:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048430.465481:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342720:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048430.465487:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513342656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048430.466776:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513342592:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048430.466804:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048430.466809:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048430.466815:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342720:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048430.466829:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513342656:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048431.105280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048431.361220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048432.129299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048432.385287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048433.153340:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048433.409287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048434.177367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048434.433274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048435.201350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048435.457260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048435.585327:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048435.585334:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048435.585338:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048435.585349:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2071 00000100:00080000:2.0:1781048435.585352:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048435.585358:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048435.585364:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513342784:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:2.0:1781048435.585368:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048435.585385:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513342912/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048435.585408:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048435.585411:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048435.585445:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513342848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048435.585455:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342912:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048435.586773:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513342784:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048435.586806:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513342848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048435.586923:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048435.586926:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048435.586929:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513342912:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048436.225373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048436.481289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048437.249335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048437.505261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048438.273328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048438.529251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048439.297343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048439.553268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048440.321351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048440.577243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048440.705373:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048440.705382:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048440.705386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048440.705399:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2076 00000100:00100000:1.0:1781048440.705402:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513342976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048440.705404:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048440.705414:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048440.705428:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048440.705445:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513343104/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048440.705472:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048440.705476:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048440.705502:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513343040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048440.705526:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343104:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048440.707453:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513343040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048440.707467:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048440.707488:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048440.707495:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343104:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048440.707496:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513342976:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781048441.345343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048441.601263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048442.369400:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048442.625257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048443.393404:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048443.649247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048444.417255:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048444.673281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048445.441343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048445.697265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048445.825400:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048445.825410:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048445.825415:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048445.825429:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2081 00000100:00080000:3.0:1781048445.825432:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048445.825441:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048445.825442:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513343168:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048445.825454:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048445.825475:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513343296/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048445.825498:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048445.825502:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048445.825536:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343296:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048445.825538:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513343232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048445.826934:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513343168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048445.826934:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513343232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048445.826943:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048445.826949:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048445.826953:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343296:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048446.465359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048446.721238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048447.489331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048447.745270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048448.513306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048448.769186:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048449.537387:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048449.793248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048450.561316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048450.817254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048450.945330:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048450.945337:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048450.945340:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048450.945352:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2086 00000100:00100000:3.0:1781048450.945353:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513343360:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048450.945356:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048450.945364:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048450.945375:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048450.945387:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c170795f x1867562513343488/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048450.945401:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048450.945404:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048450.945437:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513343424:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048450.945447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343488:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048450.946654:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513343360:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048450.946655:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513343424:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048450.946662:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048450.946666:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048450.946671:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343488:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048451.585291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048451.841237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048452.609328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048452.865261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048453.633336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048453.889249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048454.657339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048454.913187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048455.681307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048455.937195:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048456.065344:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048456.065351:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048456.065355:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048456.065365:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2091 00000100:00080000:0.0:1781048456.065368:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048456.065374:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048456.065383:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048456.065396:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513343552:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048456.065398:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513343680/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048456.065417:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048456.065420:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048456.065447:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513343616:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048456.065462:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343680:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048456.066879:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513343552:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048456.066880:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781048456.066881:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513343616:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048456.066886:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048456.066892:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343680:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048456.705337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048456.961262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048457.729363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048457.985266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048458.753343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048459.009211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048459.777300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048460.033252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048460.801344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048461.057287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048461.185331:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048461.185339:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048461.185343:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048461.185354:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2097 00000100:00080000:3.0:1781048461.185357:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048461.185359:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513343744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048461.185365:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048461.185377:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048461.185396:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513343872/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048461.185419:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048461.185422:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048461.185449:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343872:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048461.185455:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513343808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048461.186779:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513343808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048461.186780:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513343744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048461.186784:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048461.186790:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048461.186794:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513343872:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048461.825282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048462.081257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048462.849309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048463.105255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048463.873316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048464.129221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048464.897319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048465.153261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048465.921282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048466.177215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048466.305374:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048466.305398:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048466.305401:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048466.305413:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2102 00000100:00080000:0.0:1781048466.305416:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048466.305422:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048466.305434:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048466.305450:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513344064/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048466.305473:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048466.305477:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048466.305507:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513344000:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048466.305624:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781048466.305632:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513343936:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048466.305791:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344064:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00080000:3.0:1781048466.307737:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048466.307745:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048466.307752:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344064:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048466.307832:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513343936:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048466.307847:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513344000:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048466.945316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048467.201248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048467.969370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048468.225184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048468.993311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048469.249255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048470.017352:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048470.273201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048471.041347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048471.297280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048471.425299:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048471.425308:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048471.425312:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048471.425324:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2107 00000100:00100000:3.0:1781048471.425324:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513344128:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048471.425328:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048471.425337:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048471.425349:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048471.425380:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000521535b2 x1867562513344256/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048471.425402:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048471.425406:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048471.425422:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048471.425429:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513344192:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048471.425438:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344256:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1781048471.426795:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048471.426799:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513344192:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048471.426801:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048471.426809:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344256:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048471.426815:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513344128:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048472.065345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048472.321251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048473.089316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048473.345241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048474.113254:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048474.369277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048475.137325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048475.393270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048476.161306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048476.417276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048476.545357:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048476.545366:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781048476.545378:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513344320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048476.545391:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048476.545398:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2112 00000100:00080000:0.0:1781048476.545402:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048476.545408:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048476.545419:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048476.545434:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513344448/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048476.545454:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048476.545457:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048476.545474:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781048476.545494:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513344384:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048476.545519:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344448:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048476.546795:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513344384:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048476.546808:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048476.546812:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048476.546813:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513344320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048476.546818:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344448:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048477.185325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048477.441247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048478.209336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048478.465194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048479.233303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048479.489227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048480.257314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048480.513243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048481.281255:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048481.537241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048481.665321:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048481.665332:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048481.665336:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048481.665348:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2117 00000100:00100000:1.0:1781048481.665349:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513344512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048481.665353:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048481.665363:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048481.665376:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048481.665406:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002db543f2 x1867562513344640/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048481.665428:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048481.665432:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048481.665458:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513344576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048481.665465:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344640:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048481.666521:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513344512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048481.666522:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513344576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048481.666650:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048481.666653:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048481.666657:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344640:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048482.305351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048482.561262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048483.329287:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048483.585224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048484.353380:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048484.609268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048485.377360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048485.633212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048486.401334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048486.657246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048486.785356:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781048486.785358:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513344704:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048486.785363:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048486.785367:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048486.785374:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2122 00000100:00080000:2.0:1781048486.785376:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048486.785381:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048486.785392:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048486.785408:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e12224a x1867562513344832/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048486.785450:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048486.785453:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048486.785475:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513344768:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048486.785488:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344832:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048486.786967:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513344768:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048486.786968:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513344704:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048486.786976:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048486.786981:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048486.786986:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513344832:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048487.425415:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048487.681220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048488.449351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048488.705241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048489.473341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048489.729287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048490.497338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048490.753303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048491.521370:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048491.777299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048491.905386:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048491.905396:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048491.905400:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048491.905414:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2127 00000100:00080000:0.0:1781048491.905418:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781048491.905424:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513344896:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048491.905428:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048491.905444:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048491.905463:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513345024/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048491.905489:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048491.905493:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048491.905521:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513344960:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048491.905529:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345024:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048491.906966:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513344896:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048491.906971:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513344960:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048491.906973:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048491.906978:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048491.907000:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345024:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048492.545295:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048492.801286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048493.569360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048493.825289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048494.593361:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048494.849289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048495.617349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048495.873280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048496.641301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048496.897279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048497.025352:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048497.025361:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048497.025366:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048497.025379:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2132 00000100:00100000:1.0:1781048497.025380:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513345088:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048497.025399:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048497.025407:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048497.025419:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048497.025436:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513345216/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048497.025457:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048497.025461:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048497.025486:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781048497.025494:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513345152:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048497.025502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345216:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048497.026574:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513345152:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048497.026591:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781048497.026592:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513345088:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048497.026597:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048497.026602:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345216:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048497.665325:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048497.921178:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048498.689240:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048498.945224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048499.713342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048499.969216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048500.737249:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048500.993247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048501.761360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048502.017270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048502.145394:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048502.145403:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048502.145407:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048502.145415:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513345344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048502.145416:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2138 00000100:00080000:3.0:1781048502.145421:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048502.145428:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048502.145439:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048502.145454:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513345408/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048502.145472:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048502.145475:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048502.145498:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513345280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048502.145510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345408:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048502.146796:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513345280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048502.146799:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513345344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048502.146805:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048502.146809:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048502.146814:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345408:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048502.785358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048503.041285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048503.809337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048504.065253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048504.833322:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048505.089215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048505.857314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048506.113256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048506.881355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048507.137201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048507.265311:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048507.265318:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048507.265321:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048507.265331:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2143 00000100:00080000:3.0:1781048507.265334:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048507.265340:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048507.265350:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048507.265364:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513345600/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781048507.265365:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513345536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048507.265385:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048507.265388:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048507.265422:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345600:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048507.265433:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513345472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048507.266728:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513345536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048507.266743:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513345472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048507.266764:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048507.266769:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048507.266773:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345600:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048507.905385:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048508.161243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048508.929271:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048509.185212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048509.953305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048510.209225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048510.977341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048511.233243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048512.001336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048512.257219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048512.385272:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048512.385277:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048512.385280:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048512.385288:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2148 00000100:00080000:0.0:1781048512.385291:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048512.385295:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048512.385314:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048512.385324:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513345664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048512.385328:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513345792/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048512.385347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048512.385350:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048512.385366:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513345728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048512.385401:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345792:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048512.386645:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513345664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048512.386647:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513345728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048512.386665:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048512.386682:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048512.386688:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345792:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048513.025309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048513.281253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048514.049286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048514.305220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048515.073290:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048515.329249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048516.097257:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048516.353220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048517.121305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048517.377217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048517.505292:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048517.505297:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048517.505300:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048517.505308:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2153 00000100:00080000:2.0:1781048517.505311:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048517.505316:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048517.505324:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048517.505333:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513345856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048517.505338:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513345984/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048517.505352:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048517.505355:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048517.505387:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345984:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048517.505387:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513345920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048517.506459:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513345920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048517.506480:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513345856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048517.506489:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048517.506494:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048517.506499:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513345984:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048518.145263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048518.401224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048519.169285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048519.425249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048520.193228:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048520.449199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048521.217300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048521.473214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048522.241258:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048522.497216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048522.625304:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048522.625310:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048522.625314:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048522.625323:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2158 00000100:00080000:2.0:1781048522.625325:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048522.625331:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048522.625342:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048522.625346:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513346048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048522.625363:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513346176/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048522.625384:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048522.625388:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048522.625408:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513346112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048522.625435:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346176:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048522.626755:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513346112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048522.626875:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513346048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048522.626942:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048522.626946:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048522.626950:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346176:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048523.265307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048523.521242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048524.289368:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048524.545181:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048525.313297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048525.569263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048526.337315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048526.593197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048527.361294:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048527.617263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048527.745250:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048527.745255:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048527.745258:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048527.745267:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2163 00000100:00080000:2.0:1781048527.745270:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048527.745276:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048527.745284:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048527.745289:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513346240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048527.745297:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e12224a x1867562513346368/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048527.745312:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048527.745315:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048527.745333:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513346304:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048527.745353:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346368:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048527.746734:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513346304:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048527.746763:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048527.746764:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513346240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048527.746770:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048527.746778:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346368:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048528.385295:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048528.641229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048529.409357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048529.665259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048530.433311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048530.689227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048531.457311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048531.713253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048532.481291:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048532.737231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048532.865325:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048532.865353:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048532.865356:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781048532.865362:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513346432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048532.865364:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2168 00000100:00080000:0.0:1781048532.865368:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048532.865375:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048532.865385:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048532.865402:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513346560/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048532.865427:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048532.865431:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048532.865452:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513346496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048532.865469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346560:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048532.866626:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513346432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048532.866650:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513346496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048532.867013:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048532.867032:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048532.867038:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346560:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048533.505327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048533.761253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048534.529369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048534.785274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048535.553303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048535.809264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048536.577353:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048536.833262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048537.601262:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048537.857233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048537.985357:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048537.985367:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048537.985370:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048537.985383:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2173 00000100:00100000:1.0:1781048537.985383:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513346624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048537.985386:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048537.985405:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048537.985417:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048537.985431:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513346752/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048537.985453:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048537.985475:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048537.985508:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346752:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048537.985509:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513346688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048537.986658:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513346624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048537.986675:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513346688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048537.986719:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048537.986724:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048537.986728:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346752:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048538.625435:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048538.881273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048539.649308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048539.905250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048540.673329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048540.929264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048541.697321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048541.953206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048542.721331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048542.977273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048543.105385:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048543.105394:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048543.105398:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048543.105412:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2179 00000100:00100000:1.0:1781048543.105413:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513346816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048543.105417:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048543.105427:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048543.105440:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048543.105459:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513346944/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048543.105484:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048543.105488:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048543.105505:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513346880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048543.105537:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346944:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048543.106632:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513346816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048543.106666:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048543.106670:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048543.106675:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513346944:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048543.106751:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513346880:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781048543.745326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048544.001209:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048544.769323:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048545.025292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048545.793377:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048546.049276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048546.817365:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048547.073263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048547.841326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048548.097262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048548.225387:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048548.225396:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048548.225400:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048548.225413:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2184 00000100:00080000:2.0:1781048548.225418:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048548.225426:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048548.225440:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048548.225453:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513347008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048548.225459:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4afdb11 x1867562513347136/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048548.225486:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048548.225490:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048548.225523:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347136:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048548.225523:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513347072:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048548.226617:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513347008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048548.226638:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781048548.226639:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513347072:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048548.226645:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048548.226653:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347136:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048548.865347:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048549.121272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048549.889345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048550.145241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048550.913328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048551.169238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048551.937362:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048552.193266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048552.961321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048553.217274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048553.345340:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048553.345368:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048553.345372:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781048553.345380:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513347200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048553.345382:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2189 00000100:00080000:2.0:1781048553.345386:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048553.345393:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048553.345405:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048553.345424:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000521535b2 x1867562513347328/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048553.345449:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048553.345452:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048553.345479:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513347264:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048553.345489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347328:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048553.346696:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513347200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048553.346708:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513347264:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048553.346725:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048553.346729:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048553.346733:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347328:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048553.986170:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048554.241266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048555.009269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048555.265251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048556.033350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048556.289233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048557.057324:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048557.313249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048558.081363:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048558.337256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048558.465309:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048558.465315:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048558.465318:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048558.465329:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2194 00000100:00080000:1.0:1781048558.465338:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781048558.465347:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347392:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048558.465348:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048558.465357:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048558.465390:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c26c4b8c x1867562513347520/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048558.465419:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048558.465423:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048558.465460:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513347456:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048558.465520:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347520:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048558.466558:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513347456:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048558.466561:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347392:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:02020000:3.0:1781048558.466810:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:3.0:1781048558.469942:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048558.469949:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048558.469956:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347520:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:1.0:1781048559.105369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048559.361249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048560.129358:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048560.385265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048561.153299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048561.409231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048562.177315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048562.433253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048563.201328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048563.457253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048563.585383:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048563.585394:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048563.585399:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048563.585414:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2199 00000100:00080000:1.0:1781048563.585419:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048563.585428:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048563.585442:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048563.585452:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347584:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048563.585463:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008bfccf5d x1867562513347712/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048563.585490:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048563.585494:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048563.585528:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513347648:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048563.585546:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347712:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048563.586787:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347584:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048563.586802:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513347648:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048563.586831:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048563.586839:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048563.586844:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347712:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048564.225283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048564.481248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048565.249296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048565.505240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048566.273303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048566.529227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048567.297303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048567.553236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048568.321310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048568.577222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048568.705269:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048568.705275:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048568.705278:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048568.705286:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2204 00000100:00080000:1.0:1781048568.705289:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048568.705294:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781048568.705302:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347776:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781048568.705302:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048568.705315:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001cbc13b4 x1867562513347904/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048568.705333:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048568.705335:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048568.705351:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513347840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048568.705383:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347904:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048568.707081:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048568.707103:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513347840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048568.707154:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048568.707163:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048568.707169:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513347904:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048569.345331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048569.601282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048570.369355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048570.625197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048571.393283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048571.649242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048572.417251:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048572.673236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048573.441281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048573.697243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048573.825375:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048573.825380:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048573.825383:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048573.825391:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2209 00000100:00080000:1.0:1781048573.825394:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048573.825399:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048573.825407:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048573.825414:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048573.825424:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a161b523 x1867562513348096/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048573.825459:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048573.825463:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048573.825486:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513348032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048573.825502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348096:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048573.826575:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513347968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048573.826590:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513348032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048573.826607:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048573.826612:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048573.826618:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348096:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781048574.465255:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048574.721235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048575.489319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048575.745233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048576.513264:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048576.769228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781048577.537268:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048577.793248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048578.561256:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048578.817230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048578.945321:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048578.945328:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048578.945331:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048578.945339:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2214 00000100:00080000:2.0:1781048578.945342:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048578.945349:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048578.945349:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513348160:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781048578.945361:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048578.945380:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c91281b8 x1867562513348288/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048578.945403:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048578.945407:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048578.945429:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513348224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048578.945438:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348288:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048578.946552:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513348160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048578.946554:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513348224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048578.946571:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048578.946576:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048578.946596:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348288:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048579.585312:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048579.841233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048580.609272:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048580.865205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048581.633276:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048581.889241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048582.657309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048582.913192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048583.681319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048583.937243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048584.065272:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048584.065278:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048584.065281:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048584.065291:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2219 00000100:00080000:1.0:1781048584.065295:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781048584.065302:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513348352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048584.065303:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048584.065316:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048584.065335:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d7e018b1 x1867562513348480/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048584.065361:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048584.065364:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048584.065388:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513348416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048584.065408:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348480:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048584.066626:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513348352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048584.066660:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513348416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048584.066970:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048584.066974:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048584.066986:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348480:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048584.705278:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048584.961221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048585.729286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048585.985209:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048586.753296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048587.009232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048587.777305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048588.033231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048588.801310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048589.057251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048589.185297:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048589.185303:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048589.185306:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048589.185315:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2225 00000100:00080000:2.0:1781048589.185319:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048589.185325:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048589.185335:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048589.185351:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513348544:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048589.185352:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000523c4981 x1867562513348672/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048589.185381:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048589.185385:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048589.185402:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513348608:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048589.185429:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348672:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048589.186677:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513348544:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048589.186701:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048589.186707:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048589.186713:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348672:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048589.186715:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513348608:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048589.825280:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048590.081232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048590.849303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048591.105229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048591.873298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048592.129228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048592.897283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048593.153223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048593.921289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048594.177228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048594.305281:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048594.305288:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048594.305291:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048594.305300:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2230 00000100:00080000:2.0:1781048594.305303:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048594.305309:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048594.305313:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513348800:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:2.0:1781048594.305317:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048594.305329:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513348864/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048594.305345:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048594.305347:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048594.305388:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513348736:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048594.305407:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348864:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048594.306424:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513348736:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048594.306458:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513348800:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048594.312266:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048594.312274:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048594.312281:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513348864:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048594.945279:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048595.201230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048595.969270:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048596.225245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048596.993277:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048597.249241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048598.017270:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048598.273226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048599.041324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048599.297249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048599.425321:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781048599.425325:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513348928:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048599.425344:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048599.425348:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048599.425355:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2235 00000100:00080000:1.0:1781048599.425357:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048599.425362:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048599.425371:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048599.425384:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000000dbb310 x1867562513349056/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048599.425402:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048599.425405:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048599.425417:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781048599.425428:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513348992:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048599.425448:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349056:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048599.426436:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513348992:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048599.426448:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513348928:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048599.426465:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048599.426471:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048599.426476:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349056:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048600.065319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048600.321206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048601.089292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048601.345234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048602.113306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048602.369244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048603.137348:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048603.393239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048604.161281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048604.417185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048604.545320:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048604.545327:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048604.545343:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048604.545358:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2240 00000100:00080000:2.0:1781048604.545362:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048604.545367:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048604.545385:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048604.545405:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb943f69 x1867562513349248/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781048604.545405:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513349120:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048604.545432:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048604.545443:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048604.545474:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349248:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048604.545482:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513349184:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048604.547888:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513349120:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048604.547888:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513349184:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048604.547893:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048604.547901:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048604.547905:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349248:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048605.185271:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048605.441258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048606.209363:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048606.465239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048607.233351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048607.489247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048608.257352:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048608.513266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048609.281293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048609.537261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048609.665373:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048609.665382:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048609.665386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048609.665404:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2245 00000100:00100000:3.0:1781048609.665406:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513349312:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048609.665410:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048609.665419:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048609.665432:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048609.665450:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e5c8472 x1867562513349440/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048609.665492:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048609.665495:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048609.665515:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048609.665525:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513349376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048609.665533:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349440:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781048609.667034:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048609.667039:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048609.667046:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349440:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048609.667065:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513349376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048609.667088:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513349312:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781048610.305336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048610.561256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048611.329328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048611.585238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048612.353336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048612.609273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048613.377277:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048613.633266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048614.401321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048614.657254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048614.785334:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048614.785343:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048614.785347:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048614.785358:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2250 00000100:00100000:0.0:1781048614.785359:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513349504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048614.785362:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048614.785370:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048614.785379:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048614.785393:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ea0bf1d5 x1867562513349632/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048614.785411:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048614.785414:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048614.785433:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513349568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048614.785452:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349632:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048614.786747:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513349504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048614.786755:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781048614.786757:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513349568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048614.786764:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048614.786769:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349632:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048615.425277:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048615.681252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048616.449314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048616.705270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048617.473311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048617.729248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048618.497356:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048618.753273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048619.521420:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048619.777243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048619.905363:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048619.905372:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048619.905376:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048619.905384:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513349696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048619.905394:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2255 00000100:00080000:1.0:1781048619.905398:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048619.905405:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048619.905417:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048619.905437:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ce8e0ca x1867562513349824/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048619.905452:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048619.905454:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048619.905470:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513349760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048619.905482:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349824:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781048619.906767:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513349696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048619.906767:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513349760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048619.907176:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048619.907187:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048619.907197:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513349824:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048620.545296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048620.801224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048621.569298:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048621.825174:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048622.593272:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048622.849205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048623.617297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048623.873188:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048624.641351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048624.897233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048625.025264:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048625.025270:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048625.025273:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048625.025281:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2260 00000100:00080000:1.0:1781048625.025284:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048625.025288:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048625.025297:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048625.025298:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513349888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048625.025324:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513350016/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048625.025339:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048625.025341:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048625.025353:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513349952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048625.025396:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781048625.026464:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513349952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048625.026478:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513349888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048625.026573:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048625.026577:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048625.026581:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350016:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048625.665284:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048625.921234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048626.689312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048626.945222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048627.713296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048627.969188:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048628.737273:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048628.993236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048629.761345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048630.017295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048630.145374:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048630.145381:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048630.145385:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048630.145397:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2266 00000100:00100000:3.0:1781048630.145399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048630.145401:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048630.145410:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048630.145419:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048630.145436:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018a334bb x1867562513350208/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048630.145456:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048630.145459:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048630.145500:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781048630.145504:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048630.145509:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513350144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048630.146677:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513350144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048630.146687:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048630.146747:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048630.146752:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048630.146758:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350208:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048630.785307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048631.041240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048631.809359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048632.065290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048632.833275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048633.089229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048633.857345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048634.113276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048634.881287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048635.137233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048635.265335:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048635.265343:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048635.265348:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048635.265361:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2271 00000100:00100000:3.0:1781048635.265365:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048635.265366:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048635.265376:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048635.265389:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048635.265408:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb943f69 x1867562513350400/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048635.265428:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048635.265431:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048635.265461:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513350336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048635.265467:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350400:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048635.266648:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513350336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048635.266649:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048635.266655:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048635.266664:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350400:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048635.266664:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350272:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781048635.905319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048636.161204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048636.929246:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048637.185259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048637.953348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048638.209291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048638.977285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048639.233245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048640.001251:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048640.257249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048640.385269:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048640.385275:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048640.385279:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048640.385290:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2276 00000100:00080000:2.0:1781048640.385293:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048640.385299:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048640.385304:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350464:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781048640.385309:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048640.385323:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513350592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048640.385359:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048640.385362:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048640.385381:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513350528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048640.385418:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350592:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048640.386520:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350464:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048640.386521:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513350528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048640.386558:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048640.386562:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048640.386566:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350592:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048641.025338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048641.281259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048642.049301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048642.305255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048643.073360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048643.329191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048644.097329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048644.353276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048645.121352:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048645.377310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048645.505401:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048645.505410:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048645.505414:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048645.505427:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048645.505430:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2281 00000100:00080000:3.0:1781048645.505435:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048645.505457:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048645.505469:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048645.505488:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513350784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048645.505512:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048645.505520:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048645.505550:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350784:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048645.505575:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048645.505585:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513350720:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048645.506864:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048645.506869:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513350720:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048645.506871:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048645.506879:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350784:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048645.506885:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350656:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048646.145345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048646.401274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048647.169267:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048647.425291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048648.193329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048648.449291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048649.217305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048649.473268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048650.241322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048650.497300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048650.625329:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048650.625336:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048650.625339:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048650.625360:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2286 00000100:00080000:3.0:1781048650.625363:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048650.625367:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048650.625371:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048650.625404:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048650.625420:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513350976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048650.625435:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048650.625437:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048650.625461:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513350912:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048650.625479:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350976:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048650.626828:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513350912:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048650.626829:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513350848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048650.626840:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048650.626846:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048650.626852:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513350976:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048651.265312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048651.521242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048652.289269:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048652.545214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048653.313342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048653.569261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048654.337319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048654.593268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048655.361335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048655.617280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048655.745381:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048655.745390:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048655.745395:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781048655.745400:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513351040:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048655.745405:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2291 00000100:00080000:2.0:1781048655.745410:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 6s 00000100:00080000:2.0:1781048655.745418:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048655.745431:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048655.745454:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513351168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048655.745474:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048655.745477:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048655.745510:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048655.745517:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513351104:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048655.745522:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351168:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00080000:0.0:1781048655.746662:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048655.746667:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513351104:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048655.746668:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048655.746675:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351168:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048655.746685:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513351040:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048656.385293:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048656.641193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048657.409347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048657.665261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048658.433406:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048658.689322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048659.457350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048659.713295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048660.481357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048660.737260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048660.865362:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048660.865369:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048660.865387:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048660.865398:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2296 00000100:00100000:1.0:1781048660.865401:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513351296:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048660.865407:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 11s 00000100:00080000:0.0:1781048660.865414:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048660.865424:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048660.865441:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513351360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048660.865462:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048660.865467:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048660.865495:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781048660.865505:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513351232:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048660.865519:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351360:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048660.866606:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513351232:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048660.866622:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513351296:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048660.866740:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048660.866744:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048660.866747:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351360:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048661.505294:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048661.761324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048662.529255:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048662.785262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048663.553303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048663.809262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048664.577329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048664.833243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048665.601263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048665.857252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048665.985328:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048665.985335:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048665.985339:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048665.985351:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2301 00000100:00100000:1.0:1781048665.985352:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513351488:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048665.985356:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00080000:0.0:1781048665.985364:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048665.985374:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048665.985410:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513351552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048665.985433:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048665.985437:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048665.985462:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781048665.985470:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513351424:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048665.985522:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351552:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048665.986617:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513351488:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048665.986655:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513351424:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048665.986732:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048665.986738:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048665.986743:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351552:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048666.625313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048666.881294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048667.649355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048667.905365:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048668.673365:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048668.929250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048669.697351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048669.953305:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048670.721314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048670.977279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048671.105344:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048671.105354:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048671.105358:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048671.105370:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2307 00000100:00100000:3.0:1781048671.105373:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513351616:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048671.105377:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048671.105388:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048671.105402:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048671.105423:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c170795f x1867562513351744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048671.105443:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048671.105447:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048671.105473:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513351680:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048671.105479:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351744:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048671.106737:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513351680:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048671.106755:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513351616:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048671.106762:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048671.106766:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048671.106771:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351744:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048671.745257:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048672.001306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048672.769346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048673.025279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048673.793334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048674.049308:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048674.817314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048675.073259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048675.841248:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048676.097248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048676.225350:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048676.225360:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048676.225364:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048676.225379:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2312 00000100:00100000:1.0:1781048676.225387:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513351808:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048676.225400:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048676.225408:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048676.225420:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048676.225440:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513351936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048676.225465:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048676.225469:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048676.225498:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781048676.225505:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513351872:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048676.225509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351936:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048676.226618:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513351808:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048676.226627:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048676.226631:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048676.226636:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513351936:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048676.226653:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513351872:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048676.865355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048677.121309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048677.889314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048678.145263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048678.913331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048679.169274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048679.937329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048680.193222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048680.961299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048681.217273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048681.345307:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048681.345313:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048681.345317:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048681.345328:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2317 00000100:00080000:0.0:1781048681.345332:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048681.345338:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048681.345348:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048681.345363:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513352128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781048681.345375:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352000:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048681.345379:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048681.345387:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048681.345413:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352064:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048681.345431:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352128:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048681.346829:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352000:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048681.346846:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352064:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048681.346871:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048681.346876:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048681.346880:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352128:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048681.985266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048682.241186:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048683.009264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048683.265241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048684.033303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048684.289281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048685.057275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048685.313227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048686.081296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048686.337288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048686.465302:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048686.465308:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048686.465311:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048686.465320:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2322 00000100:00080000:0.0:1781048686.465323:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781048686.465327:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352192:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048686.465333:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048686.465348:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048686.465365:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513352320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048686.465382:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048686.465384:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048686.465407:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352256:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048686.465433:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352320:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048686.466811:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352256:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048686.466860:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352192:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048686.466902:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048686.466906:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048686.466911:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352320:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048687.105378:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048687.361210:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048688.129292:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048688.385246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048689.153361:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048689.409243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048690.177320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048690.433249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048691.201266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048691.457236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048691.585312:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048691.585319:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048691.585322:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048691.585332:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2327 00000100:00080000:0.0:1781048691.585335:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048691.585342:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048691.585357:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352384:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048691.585360:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048691.585391:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513352512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048691.585408:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048691.585411:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048691.585430:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352448:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048691.585500:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352512:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048691.586858:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352384:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048691.586883:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352448:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048691.586990:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048691.586995:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048691.586999:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352512:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048692.225338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048692.481254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048693.249311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048693.505198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048694.273283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048694.529229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048695.297275:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048695.553237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048696.321322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048696.577284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048696.705313:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048696.705322:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048696.705326:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048696.705340:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2332 00000100:00100000:1.0:1781048696.705342:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352576:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048696.705345:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048696.705353:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048696.705366:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048696.705385:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513352704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048696.705406:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048696.705410:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048696.705447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352704:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048696.705449:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352640:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048696.706835:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352576:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048696.706855:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513352640:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048696.706886:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048696.706891:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048696.706897:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352704:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048697.345315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048697.601226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048698.369332:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048698.625250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048699.393322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048699.649203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048700.417326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048700.673278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048701.441355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048701.697245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048701.825320:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048701.825327:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048701.825345:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048701.825357:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2337 00000100:00080000:0.0:1781048701.825360:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048701.825367:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048701.825381:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048701.825399:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352768:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048701.825404:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513352896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048701.825448:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048701.825452:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048701.825489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352896:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048701.825519:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781048701.825531:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513352832:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048701.826740:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513352832:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048701.826756:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352768:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048701.826962:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048701.826968:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048701.826974:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513352896:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048702.465302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048702.721268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048703.489318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048703.745225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048704.513271:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048704.769217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048705.537333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048705.793227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048706.561374:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048706.817242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048706.945356:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048706.945365:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048706.945369:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048706.945393:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2342 00000100:00080000:0.0:1781048706.945397:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048706.945405:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048706.945422:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352960:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048706.945427:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048706.945451:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513353088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048706.945477:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048706.945481:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048706.945511:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353024:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048706.945526:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353088:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048706.946913:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513352960:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048706.946929:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353024:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048706.947005:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048706.947010:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048706.947013:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353088:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048707.585308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048707.841253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048708.609343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048708.865215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048709.633371:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048709.889284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048710.657378:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048710.913208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048711.681331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048711.937257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048712.065348:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048712.065355:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048712.065358:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048712.065368:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2347 00000100:00080000:2.0:1781048712.065386:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781048712.065386:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513353152:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048712.065395:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048712.065403:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048712.065417:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513353280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048712.065435:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048712.065437:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048712.065455:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513353216:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048712.065472:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353280:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048712.066611:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513353152:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048712.066649:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513353216:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1781048712.066670:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781048712.070688:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048712.070696:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048712.070705:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353280:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048712.705269:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048712.961237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048713.729318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048713.985245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048714.753285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048715.009211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048715.777315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048716.033246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048716.801347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048717.057194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048717.185348:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048717.185357:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048717.185361:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048717.185371:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2353 00000100:00100000:1.0:1781048717.185372:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513353344:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048717.185376:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048717.185386:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048717.185398:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048717.185417:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513353472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048717.185452:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048717.185456:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048717.185484:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353408:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048717.185510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353472:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048717.186776:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353408:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048717.186858:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513353344:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048717.186897:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048717.186902:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048717.186907:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353472:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048717.825303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048718.081225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048718.849296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048719.105185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048719.873309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048720.129234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048720.897320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048721.153240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048721.921285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048722.177228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048722.305253:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048722.305261:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048722.305265:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781048722.305281:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513353536:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048722.305292:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2358 00000100:00080000:0.0:1781048722.305296:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048722.305302:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048722.305312:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048722.305330:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513353664/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048722.305348:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048722.305351:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048722.305371:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353600:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048722.305386:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353664:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048722.306526:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513353536:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048722.306548:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353600:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048722.306570:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048722.306575:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048722.306581:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353664:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048722.945299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048723.201232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048723.969296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048724.225229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048724.993299:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048725.249264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048726.017281:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048726.273219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048727.041307:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048727.297240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048727.425367:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048727.425372:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048727.425375:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048727.425384:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2363 00000100:00080000:2.0:1781048727.425386:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048727.425391:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048727.425399:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048727.425411:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513353728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048727.425412:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2849ba3 x1867562513353856/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048727.425430:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048727.425433:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048727.425464:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513353792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048727.425471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353856:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048727.426776:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513353792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048727.426786:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513353728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048727.426807:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048727.426811:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048727.426818:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513353856:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048728.065341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048728.321239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048729.089329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048729.345225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048730.113265:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048730.369393:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048731.137340:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048731.393280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048732.161308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048732.417215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048732.545377:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048732.545386:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048732.545390:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048732.545404:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2368 00000100:00100000:1.0:1781048732.545405:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513353920:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048732.545409:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048732.545418:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048732.545431:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048732.545466:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513354048/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048732.545492:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048732.545496:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048732.545524:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353984:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048732.545536:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354048:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048732.547007:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513353984:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048732.547065:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513353920:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048732.547102:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048732.547107:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048732.547143:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354048:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048733.185345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048733.441257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048734.209257:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048734.465221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048735.233351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048735.489250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048736.257359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048736.513238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048737.281331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048737.537274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048737.665289:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048737.665298:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048737.665303:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048737.665315:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2373 00000100:00100000:3.0:1781048737.665315:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513354112:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048737.665319:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048737.665327:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048737.665339:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048737.665353:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000015c9d1f3 x1867562513354240/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048737.665370:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048737.665373:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048737.665393:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354240:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048737.665403:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513354176:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048737.666633:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513354112:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048737.666661:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513354176:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048737.666679:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048737.666683:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048737.666687:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354240:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048738.305332:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048738.561214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048739.329330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048739.585280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048740.353312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048740.609275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048741.377308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048741.633265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048742.401313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048742.657257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048742.785357:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048742.785366:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048742.785370:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048742.785383:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2378 00000100:00080000:0.0:1781048742.785387:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781048742.785392:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513354304:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048742.785396:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048742.785421:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048742.785439:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513354432/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048742.785459:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048742.785467:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048742.785485:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354432:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048742.785518:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513354368:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048742.786895:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513354368:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048742.786895:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513354304:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048742.786926:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048742.786931:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048742.786937:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354432:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048743.425288:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048743.681281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048744.449349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048744.705263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048745.473314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048745.729273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048746.497339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048746.753267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048747.521281:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048747.777243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048747.905350:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048747.905357:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048747.905361:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781048747.905364:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513354496:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048747.905368:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2383 00000100:00080000:0.0:1781048747.905373:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048747.905379:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048747.905389:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048747.905403:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513354624/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048747.905419:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048747.905422:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048747.905448:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513354560:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048747.905462:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354624:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048747.906944:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513354496:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048747.906999:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513354560:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048747.907023:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781048747.907029:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048747.907034:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354624:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048748.545387:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048748.801277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048749.569308:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048749.825196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048750.593333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048750.849236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048751.617264:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048751.873229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048752.641345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048752.897254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048753.025285:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048753.025307:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048753.025311:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048753.025322:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2388 00000100:00080000:2.0:1781048753.025325:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781048753.025332:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513354688:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048753.025333:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048753.025356:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048753.025377:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f57815d x1867562513354816/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048753.025399:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048753.025403:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048753.025433:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354816:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048753.025448:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513354752:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048753.026677:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513354688:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048753.026810:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513354752:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048753.026955:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048753.026960:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048753.026965:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513354816:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048753.665248:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048753.921224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048754.689311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048754.945236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048755.713291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048755.969197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048756.737319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048756.993222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048757.761281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048758.017201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048758.145333:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048758.145353:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781048758.145356:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513354880:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048758.145358:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048758.145367:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2394 00000100:00080000:0.0:1781048758.145370:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781048758.145376:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048758.145386:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048758.145417:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513355008/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048758.145441:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048758.145445:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048758.145461:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048758.145467:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355008:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048758.145468:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513354944:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048758.146519:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781048758.146521:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513354944:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:2.0:1781048758.146523:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781048758.146527:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355008:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048758.146530:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513354880:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048758.785266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048759.041185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048759.809308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048760.065220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048760.833282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048761.089244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048761.857379:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048762.113237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048762.881281:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048763.137225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048763.265385:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048763.265392:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048763.265397:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048763.265408:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2399 00000100:00080000:2.0:1781048763.265413:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781048763.265419:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048763.265425:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355072:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:2.0:1781048763.265429:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048763.265447:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f57815d x1867562513355200/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048763.265482:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048763.265485:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048763.265510:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355136:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048763.265546:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355200:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048763.266795:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355072:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048763.266857:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355136:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048763.266933:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048763.266938:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048763.266943:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355200:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048763.905335:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048764.161249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048764.929330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048765.185296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048765.953343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048766.209254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048766.977280:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048767.233257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048768.001317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048768.257227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048768.385323:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048768.385331:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048768.385335:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048768.385347:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2404 00000100:00080000:3.0:1781048768.385351:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1781048768.385357:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355264:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048768.385359:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048768.385372:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048768.385386:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513355392/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048768.385404:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048768.385408:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048768.385444:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355392:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048768.385446:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355328:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048768.386786:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355264:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048768.386823:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355328:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048768.386831:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048768.386838:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048768.386845:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355392:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048769.025313:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048769.281277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048770.049353:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048770.305307:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048771.073357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048771.329267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048772.097338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048772.353274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048773.121354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048773.377254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048773.505356:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048773.505385:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048773.505390:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048773.505405:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2409 00000100:00100000:2.0:1781048773.505406:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355456:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048773.505411:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048773.505422:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048773.505442:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048773.505460:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513355584/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048773.505479:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048773.505482:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048773.505504:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355520:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048773.505537:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355584:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048773.506981:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355520:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048773.506987:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355456:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048773.507006:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048773.507012:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048773.507019:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355584:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048774.145338:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048774.401259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048775.169336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048775.425259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048776.193313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048776.449283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048777.217297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048777.473225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048778.241303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048778.497266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048778.625296:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048778.625305:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048778.625309:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048778.625322:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2414 00000100:00080000:3.0:1781048778.625327:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1781048778.625329:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355648:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048778.625337:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048778.625364:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048778.625390:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513355776/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048778.625407:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048778.625410:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048778.625433:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355712:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048778.625449:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355776:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048778.626815:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355648:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048778.626838:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355712:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048778.626848:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048778.626853:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048778.626858:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355776:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048779.265320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048779.521273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048780.289336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048780.545265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048781.313340:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048781.569282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048782.337279:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048782.593229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048783.361313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048783.617281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048783.745308:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048783.745314:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048783.745318:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048783.745330:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2419 00000100:00100000:2.0:1781048783.745334:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048783.745334:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048783.745342:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048783.745352:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048783.745366:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513355968/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048783.745384:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048783.745386:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048783.745428:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355968:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048783.745433:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048783.746784:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513355904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048783.746814:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513355840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048783.746824:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048783.746829:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048783.746841:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513355968:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048784.385275:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048784.641256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048785.409260:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048785.665257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048786.433258:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048786.689227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048787.457302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048787.713256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048788.481282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048788.737184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048788.865292:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048788.865299:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048788.865303:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781048788.865316:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356032:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048788.865316:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2424 00000100:00080000:3.0:1781048788.865321:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048788.865330:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048788.865343:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048788.865360:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513356160/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048788.865378:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048788.865381:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048788.865413:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356160:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048788.865418:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356096:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048788.866605:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356096:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048788.866661:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356032:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048788.866685:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048788.866690:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048788.866694:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356160:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048789.505340:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048789.761262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048790.529258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048790.785187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048791.553317:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048791.809256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048792.577313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048792.833220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048793.601361:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048793.857263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048793.985320:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048793.985327:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048793.985330:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048793.985342:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2429 00000100:00080000:3.0:1781048793.985346:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1781048793.985355:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356224:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048793.985355:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048793.985370:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048793.985389:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513356352/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048793.985413:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048793.985416:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048793.985440:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356288:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048793.985466:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356352:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048793.986660:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356288:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048793.986661:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356224:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048793.986684:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048793.986689:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048793.986695:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356352:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048794.625325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048794.881194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048795.649292:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048795.905273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048796.673340:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048796.929261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048797.697301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048797.953246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048798.721341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048798.977277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048799.105318:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048799.105327:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048799.105334:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048799.105347:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2435 00000100:00100000:2.0:1781048799.105348:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048799.105352:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781048799.105360:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048799.105388:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048799.105406:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513356544/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048799.105428:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048799.105432:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048799.105456:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048799.105480:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356544:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048799.106683:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048799.106705:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048799.106723:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048799.106727:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048799.106732:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356544:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048799.745302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048800.001193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048800.769320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048801.025177:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048801.793278:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048802.049255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048802.817343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048803.073259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048803.841263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048804.097263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048804.225295:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048804.225303:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048804.225307:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048804.225319:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2440 00000100:00080000:3.0:1781048804.225323:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048804.225330:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048804.225340:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356608:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048804.225341:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048804.225364:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513356736/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048804.225389:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048804.225393:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048804.225423:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356736:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048804.225435:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048804.226721:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048804.226729:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048804.226987:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048804.226992:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048804.226997:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356736:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048804.865355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048805.121264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048805.889308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048806.145222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048806.913328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048807.169285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048807.937287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048808.193212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048808.961330:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048809.217281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048809.345277:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048809.345285:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048809.345289:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048809.345301:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2445 00000100:00080000:3.0:1781048809.345305:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048809.345311:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048809.345322:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356800:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781048809.345323:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048809.345340:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513356928/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048809.345360:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048809.345365:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048809.345390:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048809.345420:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356928:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048809.346730:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356800:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048809.346873:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513356864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048809.346908:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048809.346913:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048809.346937:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513356928:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048809.985357:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048810.241226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048811.009312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048811.265286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048812.033290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048812.289224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048813.057343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048813.313251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048814.081330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048814.337267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048814.465329:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048814.465336:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048814.465339:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048814.465350:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2450 00000100:00080000:2.0:1781048814.465353:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781048814.465354:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048814.465361:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048814.465384:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048814.465399:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000015c9d1f3 x1867562513357120/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048814.465416:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048814.465419:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048814.465455:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513357056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781048814.465460:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357120:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048814.466690:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513356992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048814.466695:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513357056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781048814.466708:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048814.466713:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048814.466718:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357120:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048815.105346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048815.361248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048816.129320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048816.385177:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048817.153336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048817.409244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048818.177403:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048818.433256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048819.201267:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048819.457243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048819.585364:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048819.585371:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048819.585375:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048819.585385:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2455 00000100:00080000:3.0:1781048819.585388:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048819.585392:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513357184:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048819.585396:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048819.585407:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048819.585420:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513357312/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048819.585433:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048819.585435:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048819.585454:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513357248:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048819.585483:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357312:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00080000:1.0:1781048819.586862:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781048819.586864:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513357184:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:1.0:1781048819.586867:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048819.586871:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357312:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048819.586908:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513357248:192.168.201.144@tcp:400:kworker/u8:1.0 00010000:00010000:3.0:1781048820.225257:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048820.481224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048821.249307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048821.505248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048822.273323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048822.529308:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048823.297255:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048823.553246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048824.321345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048824.577318:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048824.705323:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048824.705329:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048824.705352:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048824.705364:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2460 00000100:00080000:0.0:1781048824.705369:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048824.705375:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048824.705385:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048824.705399:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357376:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048824.705400:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dd1a56a6 x1867562513357504/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048824.705430:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048824.705433:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048824.705453:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513357440:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048824.705462:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357504:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048824.706768:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357376:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048824.706790:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513357440:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048824.706822:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048824.706828:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048824.706832:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357504:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048825.345263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048825.601204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048826.369304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048826.625243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048827.393263:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048827.649208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048828.417302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048828.673231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048829.441279:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048829.697218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048829.825277:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048829.825282:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048829.825285:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048829.825292:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2465 00000100:00080000:0.0:1781048829.825295:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048829.825301:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048829.825309:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048829.825320:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513357696/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781048829.825356:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357568:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048829.825358:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048829.825370:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048829.825398:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513357632:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048829.825407:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357696:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048829.826763:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357568:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048829.826770:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513357632:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048829.826789:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048829.826794:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048829.826799:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357696:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048830.465254:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048830.721241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048831.489287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048831.745224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048832.513263:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048832.769245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048833.537314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048833.793218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048834.561288:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048834.817216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048834.945244:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048834.945251:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048834.945255:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048834.945265:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2470 00000100:00100000:1.0:1781048834.945265:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357760:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048834.945270:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048834.945278:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048834.945288:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048834.945306:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002db543f2 x1867562513357888/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048834.945328:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048834.945332:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048834.945352:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513357824:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048834.945379:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357888:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048834.946429:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357760:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048834.946430:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513357824:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048834.946531:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048834.946536:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048834.946541:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513357888:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048835.585278:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048835.841246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048836.609272:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048836.865257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048837.633311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048837.889240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048838.657279:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048838.913320:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048839.681284:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048839.937235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048840.065295:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048840.065300:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048840.065304:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048840.065313:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2475 00000100:00080000:0.0:1781048840.065316:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048840.065320:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048840.065329:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048840.065341:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357952:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048840.065347:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513358080/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048840.065368:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048840.065372:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048840.065390:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358016:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048840.065403:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358080:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048840.066581:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358016:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048840.066595:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513357952:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048840.066726:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048840.066729:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048840.066733:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358080:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048840.705246:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048840.961215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048841.729283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048841.985242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048842.753272:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048843.009208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048843.777269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048844.033248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048844.801268:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048845.057204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048845.185271:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048845.185292:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048845.185295:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781048845.185299:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513358144:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048845.185302:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2481 00000100:00080000:0.0:1781048845.185305:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048845.185311:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048845.185319:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048845.185335:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513358272/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048845.185356:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048845.185360:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048845.185383:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358208:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048845.185413:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358272:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048845.186745:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513358144:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048845.186763:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358208:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048845.186844:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048845.186848:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048845.186851:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358272:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048845.825292:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048846.081224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048846.849250:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048847.105211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048847.873259:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048848.129235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048848.897242:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048849.153190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048849.921292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048850.177218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048850.305266:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048850.305274:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048850.305277:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048850.305289:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2486 00000100:00080000:0.0:1781048850.305292:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048850.305299:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048850.305309:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048850.305309:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513358336:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048850.305347:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513358464/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048850.305371:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048850.305375:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048850.305393:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358400:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048850.305453:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358464:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:0.0:1781048850.306703:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358400:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:1.0:1781048850.306727:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513358336:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048850.306827:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048850.306833:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048850.306839:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358464:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048850.945305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048851.201176:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048851.969256:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048852.225226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048852.993287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048853.249191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048854.017308:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048854.273234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781048854.401384:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513358528:192.168.201.144@tcp:17:kworker/2:0.0 00000100:00100000:3.0:1781048854.402728:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513358528:192.168.201.144@tcp:17:kworker/2:0.0 00010000:00010000:3.0:1781048855.041246:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048855.297198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048855.425367:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048855.425377:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048855.425381:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048855.425393:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2491 00000100:00080000:0.0:1781048855.425397:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048855.425405:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781048855.425407:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513358592:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:0.0:1781048855.425416:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048855.425432:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513358720/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781048855.425450:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048855.425453:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048855.425475:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048855.425491:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358720:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:1.0:1781048855.426788:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513358592:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048855.426824:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513358656:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048855.426828:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048855.426834:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048855.426841:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358720:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048856.065284:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048856.321268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048857.089349:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048857.345232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048858.113311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048858.369256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048859.137351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048859.393261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048860.161291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048860.417277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048860.545349:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048860.545356:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048860.545360:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048860.545371:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2496 00000100:00080000:3.0:1781048860.545374:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048860.545406:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513358784:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048860.545412:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048860.545424:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048860.545439:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513358912/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048860.545459:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048860.545462:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048860.545497:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358912:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:3.0:1781048860.545503:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513358848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048860.546882:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513358848:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048860.546997:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513358784:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048860.547068:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048860.547072:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048860.547076:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513358912:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048861.185356:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048861.441258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048862.209253:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048862.465236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048863.233369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048863.489257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048864.257316:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048864.513202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048865.281278:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048865.537248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048865.665288:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048865.665297:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048865.665302:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048865.665316:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2501 00000100:00080000:3.0:1781048865.665320:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048865.665327:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513358976:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048865.665330:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048865.665346:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048865.665365:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513359104/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048865.665386:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048865.665389:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048865.665409:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359040:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048865.665452:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359104:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048865.666747:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513358976:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:3.0:1781048865.666822:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359040:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:02020000:0.0:1781048865.666899:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781048865.670332:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048865.670338:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048865.670347:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359104:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048866.305307:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048866.561225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048867.329329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048867.585237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048868.353302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048868.609257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048869.377325:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048869.633262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048870.401294:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048870.657271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048870.785399:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048870.785408:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048870.785412:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048870.785426:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2506 00000100:00080000:3.0:1781048870.785430:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781048870.785432:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359168:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:3.0:1781048870.785441:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048870.785455:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048870.785474:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513359296/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048870.785497:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048870.785501:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048870.785523:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359232:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048870.785546:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359296:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048870.786824:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359168:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048870.786881:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359232:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048870.786947:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048870.786952:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048870.786958:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359296:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048871.425301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048871.681254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048872.449355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048872.705198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048873.473317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048873.729325:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048874.497337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048874.753313:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048875.521291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048875.777267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048875.905344:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781048875.905344:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781048875.905357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048875.905361:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048875.905366:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2511 00000100:00080000:2.0:1781048875.905369:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048875.905374:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048875.905383:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048875.905412:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c91281b8 x1867562513359488/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048875.905431:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048875.905434:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048875.905452:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048875.905476:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359488:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048875.906763:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781048875.906786:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048875.906789:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048875.906795:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048875.906802:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359488:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048876.545356:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048876.801260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048877.569990:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048877.825265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048878.593336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048878.849270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048879.617286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048879.873255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048880.641311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048880.897282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048881.025350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048881.025358:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048881.025362:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048881.025373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2516 00000100:00100000:2.0:1781048881.025374:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048881.025377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048881.025386:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048881.025397:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048881.025414:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513359680/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048881.025432:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048881.025435:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048881.025470:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359680:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781048881.025480:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048881.027003:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513359616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048881.027037:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048881.027077:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048881.027080:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048881.027084:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359680:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048881.665302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048881.921224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048882.689328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048882.945250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048883.713292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048883.969233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048884.737324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048884.993238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048885.761279:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048886.017219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048886.145310:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048886.145318:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048886.145322:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048886.145338:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2522 00000100:00100000:2.0:1781048886.145341:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781048886.145346:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048886.145370:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048886.145380:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048886.145398:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513359872/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048886.145419:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048886.145423:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048886.145438:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048886.145445:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513359808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781048886.145469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359872:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781048886.146879:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513359808:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781048886.146889:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781048886.146927:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048886.146932:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048886.146937:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513359872:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781048886.785295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048887.041251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048887.809254:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048888.065201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048888.833287:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048889.089225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048889.857306:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048890.113228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048890.881276:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048891.137257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048891.265264:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048891.265270:0:36:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048891.265278:0:36:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048891.265287:0:36:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2527 00000100:00080000:3.0:1781048891.265290:0:36:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048891.265295:0:36:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781048891.265301:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359936:192.168.201.144@tcp:400:kworker/u8:1.0 00000008:00100000:3.0:1781048891.265304:0:36:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048891.265317:0:36:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000048c6b8a x1867562513360064/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048891.265330:0:36:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048891.265333:0:36:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048891.265350:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513360000:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:0.0:1781048891.265364:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360064:192.168.201.144@tcp:8:kworker/u8:1.0 00000100:00100000:2.0:1781048891.266710:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513360000:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00100000:2.0:1781048891.266746:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513359936:192.168.201.144@tcp:400:kworker/u8:1.0 00000100:00080000:0.0:1781048891.266777:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048891.266781:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048891.266786:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360064:192.168.201.144@tcp:8:kworker/u8:1.0 00010000:00010000:3.0:1781048891.905306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048892.161175:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048892.929334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048893.185259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048893.953333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048894.209242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048894.977286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048895.233249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048896.001302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048896.257226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781048896.385300:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1781048896.385336:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048896.385341:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1781048896.385343:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513360128:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1781048896.385344:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048896.385351:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2532 00000100:00080000:3.0:1781048896.385354:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048896.385358:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048896.385362:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048896.385368:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513360192:192.168.201.144@tcp:400:kworker/u8:2.0 00000008:00100000:2.0:1781048896.385448:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048896.385463:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513360256/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048896.385478:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048896.385481:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048896.385504:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360256:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048896.386843:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513360128:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1781048896.386863:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513360192:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048896.386870:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048896.386885:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048896.386893:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360256:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048897.025250:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048897.281241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048898.049298:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048898.305221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048899.073238:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048899.329224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048900.097270:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048900.353237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048901.121292:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048901.377212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781048901.505309:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048901.505333:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513360320:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1781048901.505373:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048901.505379:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048901.505382:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048901.505388:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2537 00000100:00080000:3.0:1781048901.505391:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048901.505396:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048901.505405:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048901.505427:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513360384:192.168.201.144@tcp:400:kworker/u8:2.0 00000008:00100000:2.0:1781048901.505444:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048901.505465:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000521535b2 x1867562513360448/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048901.505487:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048901.505490:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048901.505530:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360448:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048901.506729:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513360384:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048901.506806:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048901.506811:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048901.506814:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513360320:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048901.506818:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360448:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048902.145319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048902.401243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048903.169285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048903.425205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048904.193331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048904.449238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048905.217330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048905.473242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048906.241290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048906.497263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048906.625282:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048906.625288:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048906.625291:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048906.625300:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2542 00000100:00080000:2.0:1781048906.625303:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048906.625308:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048906.625316:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048906.625318:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513360512:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048906.625345:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513360640/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048906.625360:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048906.625362:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048906.625391:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513360576:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048906.625396:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360640:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1781048906.627172:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048906.627179:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048906.627186:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360640:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1781048906.627248:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513360512:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048906.627258:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513360576:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048907.265308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048907.521319:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048908.289346:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048908.545232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048909.313366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048909.569460:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048910.337727:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048910.593249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048911.361390:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048911.618211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781048911.745379:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513360704:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048911.745539:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781048911.745547:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513360768:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1781048911.746062:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048911.746068:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048911.746071:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048911.746077:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2547 00000100:00080000:3.0:1781048911.746080:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048911.746086:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048911.746096:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048911.746113:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000048c6b8a x1867562513360832/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781048911.746199:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360832:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00080000:3.0:1781048911.746274:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048911.746278:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781048911.748473:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781048911.748480:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513360768:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048911.748483:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048911.748491:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513360832:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1781048911.748499:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513360704:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048912.385339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048912.641263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048913.409315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048913.665283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048914.433366:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048914.689301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048915.457615:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781048915.713325:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048916.481380:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048916.737642:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048916.865363:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048916.865370:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048916.865374:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048916.865384:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2552 00000100:00080000:1.0:1781048916.865388:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048916.865395:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048916.865405:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048916.865422:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d7e018b1 x1867562513361024/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781048916.865445:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513360896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781048916.865481:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361024:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781048916.865600:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1781048916.865604:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048916.865610:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781048916.865613:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513360960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781048916.868932:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513360896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781048916.868992:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781048916.868997:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513360960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781048916.869002:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048916.869012:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361024:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781048917.507093:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048917.761273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048918.529345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048918.785250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048919.553367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048919.809248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048920.579949:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048920.835254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048921.601355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048921.857208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048921.985360:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048921.985367:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048921.985370:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048921.985380:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2557 00000100:00080000:2.0:1781048921.985383:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048921.985389:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048921.985399:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048921.985443:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513361216/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048921.985462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048921.985465:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048921.985490:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513361152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781048921.985606:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048921.985612:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513361088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781048921.985650:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361216:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781048921.987790:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513361088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781048921.987979:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048921.987988:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048921.987996:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361216:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781048921.988547:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513361152:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781048922.625766:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048922.881211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048923.649423:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048923.905271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048924.674053:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048924.930752:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048925.698303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048925.953324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048926.721853:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048926.977253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048927.108741:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048927.108767:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048927.108771:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048927.108783:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2563 00000100:00080000:3.0:1781048927.108787:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048927.108793:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048927.108806:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048927.108927:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513361408/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048927.108943:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048927.108946:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048927.108975:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513361344:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1781048927.109084:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048927.109089:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513361280:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048927.109361:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361408:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048927.115199:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513361280:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048927.115225:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513361344:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048927.122417:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048927.122451:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048927.122473:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361408:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048927.745354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048928.001239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048928.773690:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048929.025240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048929.793399:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048930.049315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048930.819314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048931.073265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048931.848554:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048932.097297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048932.226823:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048932.226830:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048932.226834:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781048932.226844:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2568 00000100:00080000:1.0:1781048932.226848:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048932.226855:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048932.226865:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048932.226882:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513361600/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048932.226911:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048932.226915:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781048932.226941:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513361536:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781048932.226966:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361600:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781048932.227038:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513361472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781048932.240878:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048932.240891:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048932.240903:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361600:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781048932.240971:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513361536:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781048932.241010:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513361472:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781048932.870391:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048933.122614:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048933.892770:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048934.145240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048934.919482:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048935.169272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048935.937399:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048936.193280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048936.963414:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048937.217273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048937.346437:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048937.346446:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048937.346450:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048937.346462:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2573 00000100:00080000:2.0:1781048937.346465:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048937.346471:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048937.346482:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048937.346497:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084e7220c x1867562513361792/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048937.346521:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048937.346524:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048937.346552:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513361728:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781048937.346644:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048937.346650:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513361664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781048937.346880:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361792:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1781048937.353337:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048937.353351:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048937.353361:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361792:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781048937.353839:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513361664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781048937.353877:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513361728:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781048937.985636:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048938.241296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048939.009351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048939.265918:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048940.033525:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048940.289272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048941.058168:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048941.314641:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048942.081286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048942.337258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048942.465999:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048942.466007:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048942.466010:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048942.466047:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2578 00000100:00080000:2.0:1781048942.466050:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048942.466057:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048942.466068:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048942.466086:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513361984/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048942.466114:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048942.466117:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048942.466221:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513361856:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1781048942.466281:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361984:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1781048942.468153:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048942.468169:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513361920:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1781048942.470151:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048942.470164:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048942.470174:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513361984:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1781048942.478658:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513361856:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048942.486620:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513361920:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048943.105384:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048943.364611:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048944.129457:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048944.385775:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048945.155815:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048945.409408:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048946.178539:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048946.433251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048947.201999:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048947.457245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048947.588402:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048947.588411:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048947.588414:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048947.588425:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2583 00000100:00080000:3.0:1781048947.588429:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048947.588436:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048947.588446:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048947.588463:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000288f7157 x1867562513362176/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048947.588510:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048947.588513:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048947.588541:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513362112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781048947.588632:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048947.588648:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513362048:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781048947.588742:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362176:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781048947.592026:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513362112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781048947.592331:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048947.592336:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048947.592359:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362176:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781048947.596079:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513362048:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781048948.225644:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048948.481278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048949.250198:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048949.505252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048950.274315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048950.529316:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048951.297824:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048951.553512:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048952.321333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781048952.577467:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781048952.705337:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513362240:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1781048952.709072:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781048952.709092:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513362304:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1781048952.709605:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048952.709611:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048952.709614:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048952.709618:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2588 00000100:00080000:2.0:1781048952.709621:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048952.709640:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048952.709649:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048952.709671:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb943f69 x1867562513362368/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781048952.709740:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362368:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1781048952.709818:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048952.709821:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048952.714105:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513362304:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1781048952.714118:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513362240:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1781048952.719654:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048952.719663:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048952.719671:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362368:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048953.345346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048953.602928:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048954.369582:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048954.625305:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048955.393749:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048955.650336:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048956.417372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048956.673905:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048957.441334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048957.697350:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781048957.825434:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048957.825443:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781048957.825446:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781048957.825563:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513362432:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048957.825669:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781048957.825692:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513362496:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:1.0:1781048957.825963:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2593 00000100:00080000:1.0:1781048957.825966:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781048957.825972:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781048957.825982:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781048957.826000:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513362560/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781048957.826018:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781048957.826021:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048957.826192:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362560:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1781048957.831751:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513362496:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048957.831788:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513362432:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:3.0:1781048957.833550:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781048957.833561:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781048957.833571:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362560:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048958.466045:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048958.721282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048959.490266:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048959.745343:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048960.513377:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048960.769527:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048961.537635:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048961.793299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048962.562925:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048962.817296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781048962.945444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048962.945452:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781048962.945457:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781048962.945470:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2598 00000100:00080000:3.0:1781048962.945474:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781048962.945481:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781048962.945492:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781048962.945508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513362752/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781048962.945528:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781048962.945531:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048962.945558:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513362688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781048962.945655:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781048962.945690:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513362624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781048962.946001:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362752:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781048962.951886:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513362624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781048962.951919:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513362688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781048962.955191:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048962.955205:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048962.955216:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362752:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781048963.586287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048963.841890:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048964.612806:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048964.865478:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048965.633729:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048965.889721:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048966.657980:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048966.913416:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048967.681721:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048967.937732:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048968.067681:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048968.067691:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048968.067694:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048968.067706:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2603 00000100:00080000:2.0:1781048968.067709:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048968.067715:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048968.067725:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048968.067744:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513362944/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048968.067771:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048968.067775:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048968.067804:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513362880:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048968.067895:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048968.067902:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513362816:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1781048968.068290:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362944:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1781048968.073828:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048968.073841:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048968.073850:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513362944:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048968.076506:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513362880:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048968.076535:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513362816:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048968.708739:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048968.962874:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048969.729385:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048969.985264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048970.753907:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048971.009273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048971.777368:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048972.033253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048972.804336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048973.059753:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048973.185413:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048973.185589:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048973.185593:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048973.185604:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2609 00000100:00080000:2.0:1781048973.185607:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048973.185614:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048973.185624:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048973.185641:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084e7220c x1867562513363136/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048973.185668:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048973.185671:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048973.185698:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363072:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048973.185789:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048973.185794:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513363008:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1781048973.185890:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363136:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00080000:1.0:1781048973.190931:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781048973.190946:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048973.190957:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363136:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048973.191667:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513363008:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048973.191686:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363072:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048973.825296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048974.081269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048974.851359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048975.105214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048975.873336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048976.129262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048976.899060:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048977.153389:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048977.922948:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048978.177256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781048978.306008:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048978.306015:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781048978.306020:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781048978.306031:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2614 00000100:00080000:0.0:1781048978.306035:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781048978.306042:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781048978.306056:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781048978.306078:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513363328/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781048978.306082:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513363200:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048978.306117:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781048978.306121:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781048978.306293:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363328:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1781048978.306306:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781048978.306319:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513363264:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:1.0:1781048978.308653:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513363200:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048978.308682:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048978.308693:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781048978.308701:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513363264:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048978.308704:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363328:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048978.947046:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048979.201405:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048979.969423:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048980.225434:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048980.994079:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048981.249271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048982.018807:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048982.273262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048983.041342:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048983.298720:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048983.425400:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048983.425409:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048983.425427:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781048983.425435:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513363392:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1781048983.425436:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2619 00000100:00080000:2.0:1781048983.425441:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048983.425449:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048983.425459:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048983.425474:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513363520/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048983.425491:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048983.425493:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048983.425510:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363456:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048983.425528:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363520:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1781048983.427763:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048983.427770:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048983.427776:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363520:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048983.429400:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363456:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048983.429452:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513363392:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048984.065746:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048984.321379:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048985.089343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048985.347363:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048986.113378:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048986.370396:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048987.137263:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048987.393304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048988.164287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048988.417304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048988.545358:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048988.545366:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048988.545369:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048988.545380:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2624 00000100:00080000:2.0:1781048988.545383:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048988.545389:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048988.545400:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048988.545416:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513363712/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048988.545442:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048988.545463:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048988.545494:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363648:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048988.545600:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781048988.545606:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513363584:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048988.545941:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363712:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048988.547177:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363648:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:2.0:1781048988.547213:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513363584:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048988.547256:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048988.547260:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048988.547264:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363712:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048989.185345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048989.441495:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048990.209359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048990.465264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048991.233333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048991.489242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048992.257464:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048992.513240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048993.281363:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048993.537258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048993.665308:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048993.665317:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048993.665321:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048993.665332:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2629 00000100:00080000:2.0:1781048993.665336:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048993.665345:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781048993.665346:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513363776:192.168.201.144@tcp:400:kworker/u8:2.0 00000008:00100000:2.0:1781048993.665355:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048993.665374:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000523c4981 x1867562513363904/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781048993.665401:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048993.665404:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048993.665419:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363840:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048993.665499:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363904:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048993.666754:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513363840:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:3.0:1781048993.666757:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513363776:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048993.666767:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048993.666773:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048993.666779:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513363904:192.168.201.144@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1781048994.305557:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048994.561261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048995.329412:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048995.585221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048996.355160:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048996.609258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048997.377330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048997.633300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781048998.401348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048998.657263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781048998.785799:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048998.785822:0:20035:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781048998.785826:0:20035:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781048998.785837:0:20035:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2634 00000100:00080000:2.0:1781048998.785841:0:20035:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781048998.785848:0:20035:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781048998.785860:0:20035:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781048998.785879:0:20035:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513364096/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781048998.785885:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513363968:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:2.0:1781048998.785908:0:20035:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781048998.785912:0:20035:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781048998.785934:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513364032:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00100000:0.0:1781048998.786695:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364096:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048998.791189:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513363968:192.168.201.144@tcp:400:kworker/u8:2.0 00000100:00080000:0.0:1781048998.791355:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781048998.791361:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781048998.791366:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364096:192.168.201.144@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1781048998.793468:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513364032:192.168.201.144@tcp:400:kworker/u8:2.0 00010000:00010000:3.0:1781048999.425337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781048999.681230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049000.449313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049000.705250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049001.473837:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049001.729272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049002.497369:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049002.753250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049003.521410:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049003.777266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781049003.905407:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513364160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049003.905562:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049003.905569:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513364224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049003.906052:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049003.906058:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049003.906062:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049003.906068:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2639 00000100:00080000:3.0:1781049003.906071:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049003.906077:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049003.906087:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049003.906106:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513364288/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049003.906156:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049003.906160:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049003.906257:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364288:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049003.907840:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513364160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049003.907869:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513364224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049003.910870:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049003.910879:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049003.910887:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364288:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049004.545568:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049004.801263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049005.569344:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049005.825253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049006.594211:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049006.849257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049007.618351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049007.873979:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049008.641355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049008.899958:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049009.025347:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049009.025357:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049009.025360:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049009.025371:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2644 00000100:00080000:3.0:1781049009.025374:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049009.025380:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049009.025399:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049009.025415:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513364480/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049009.025436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049009.025439:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049009.025466:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513364416:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049009.025479:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364480:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049009.025710:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513364352:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049009.029221:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513364352:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049009.029234:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049009.029240:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049009.029260:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364480:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049009.031231:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513364416:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049009.665359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049009.921255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049010.689342:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049010.945318:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049011.713510:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049011.969247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049012.737308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049012.993624:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049013.761383:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049014.017252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049014.145275:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049014.145281:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049014.145285:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049014.145294:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2650 00000100:00080000:3.0:1781049014.145296:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049014.145302:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049014.145310:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049014.145316:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513364608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049014.145324:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513364672/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781049014.145381:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364672:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049014.145420:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:3.0:1781049014.145437:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049014.145441:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049014.145444:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513364544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049014.149331:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513364544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049014.149412:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513364608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049014.153087:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049014.153099:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049014.153109:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364672:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049014.785421:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049015.041213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049015.809473:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049016.065528:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049016.833334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049017.089229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049017.857352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049018.113258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049018.881358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049019.137200:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049019.265336:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049019.265345:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049019.265348:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049019.265359:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2655 00000100:00080000:3.0:1781049019.265363:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049019.265369:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049019.265380:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049019.265414:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513364864/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049019.265441:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049019.265445:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049019.265539:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513364736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049019.265725:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364864:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049019.265793:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049019.265801:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513364800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049019.272564:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513364736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049019.272788:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513364800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:02020000:0.0:1781049019.275380:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781049019.283424:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049019.283436:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049019.283447:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513364864:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049019.905283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049020.161774:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049020.929419:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049021.188771:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049021.953396:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049022.209250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049022.977392:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049023.233254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049024.002036:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049024.257754:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049024.385366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049024.385376:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049024.385379:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049024.385391:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2660 00000100:00080000:1.0:1781049024.385395:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781049024.385402:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513364928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049024.385426:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049024.385436:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049024.385452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008bfccf5d x1867562513365056/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049024.385477:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049024.385481:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049024.385501:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513364992:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049024.385873:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365056:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049024.389565:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513364992:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049024.390832:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513364928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049024.391042:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049024.391047:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049024.391052:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365056:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049025.025356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049025.281264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049026.049298:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049026.305258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049027.073715:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049027.329217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049028.097709:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049028.353692:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049029.121315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049029.377270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049029.505708:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049029.505716:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049029.505720:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049029.505728:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2665 00000100:00080000:3.0:1781049029.505733:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049029.505743:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781049029.505746:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365120:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781049029.505754:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049029.505774:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513365248/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049029.505807:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049029.505811:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049029.505831:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049029.505896:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365248:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049029.509246:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049029.509291:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049029.510843:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049029.510852:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049029.510859:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365248:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049030.145521:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049030.401233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049031.169324:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049031.425245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049032.193319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049032.449249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049033.217293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049033.473229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049034.241635:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049034.497259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049034.625339:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049034.625347:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049034.625351:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049034.625353:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049034.625373:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2670 00000100:00080000:3.0:1781049034.625376:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049034.625382:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049034.625392:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049034.625409:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513365440/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049034.625434:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049034.625437:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049034.625454:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049034.625580:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365440:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049034.627726:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049034.627983:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049034.633170:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049034.633183:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049034.633193:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365440:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049035.265728:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049035.521271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049036.289354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049036.545294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049037.313559:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049037.569278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049038.339541:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049038.593258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049039.361311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049039.617907:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049039.745364:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049039.745373:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049039.745377:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049039.745388:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2675 00000100:00080000:2.0:1781049039.745392:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781049039.745393:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049039.745400:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049039.745412:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049039.745429:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000521535b2 x1867562513365632/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049039.745456:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049039.745460:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049039.745487:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049039.745808:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365632:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049039.749339:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049039.751859:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049039.751871:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049039.751881:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365632:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049039.752946:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365568:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049040.385365:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049040.641315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049041.411691:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049041.665250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049042.433971:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049042.689264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049043.457883:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049043.713248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049044.481347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049044.737259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049044.865544:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049044.865552:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049044.865555:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049044.865566:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2680 00000100:00080000:3.0:1781049044.865570:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049044.865576:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049044.865587:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049044.865603:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bc445f2a x1867562513365824/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049044.865628:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049044.865632:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049044.865700:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049044.865737:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365824:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049044.865806:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781049044.865833:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513365696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049044.869833:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513365696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049044.869858:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049044.869952:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049044.869958:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049044.869962:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513365824:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049045.505334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049045.761285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049046.529327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049046.785263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049047.554821:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049047.809352:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049048.578410:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049048.834484:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049049.601480:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049049.857316:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049049.985356:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049049.985366:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049049.985370:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049049.985399:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2685 00000100:00080000:3.0:1781049049.985403:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049049.985409:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049049.985419:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049049.985438:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bc445f2a x1867562513366016/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049049.985460:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049049.985463:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049049.985484:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049049.985498:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049049.985603:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049049.990874:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049049.990897:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049049.990907:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049049.990913:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513365888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049049.990969:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513365952:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049050.625398:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049050.881248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049051.649686:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049051.905270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049052.675275:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049052.929263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049053.699081:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049053.953212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049054.721817:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049054.977225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049055.105477:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049055.105485:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049055.105489:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049055.105500:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2691 00000100:00100000:3.0:1781049055.105502:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513366080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049055.105506:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049055.105515:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049055.105527:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049055.105543:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2849ba3 x1867562513366208/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781049055.105610:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049055.105618:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513366144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049055.105671:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781049055.105732:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049055.105737:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781049055.111109:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049055.111122:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049055.111188:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049055.112117:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513366144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049055.112168:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513366080:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049055.745322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049056.001242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049056.769313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049057.026145:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049057.793330:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049058.049276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049058.817498:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049059.073542:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049059.841458:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049060.102098:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049060.225374:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049060.225381:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049060.225399:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049060.225408:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2696 00000100:00080000:1.0:1781049060.225411:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049060.225416:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049060.225421:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513366272:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781049060.225425:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049060.225498:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c26c4b8c x1867562513366400/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781049060.225527:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049060.225534:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513366336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049060.225552:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366400:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1781049060.225607:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049060.225611:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049060.227559:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513366336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049060.227573:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513366272:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049060.227685:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049060.227689:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049060.227693:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366400:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049060.865357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049061.121982:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049061.889318:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049062.145239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049062.913431:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049063.169222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049063.937607:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049064.193269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049064.961591:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049065.219950:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049065.345933:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049065.345943:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1781049065.346052:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513366464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049065.346169:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049065.346175:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513366528:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049065.346664:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049065.346671:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2701 00000100:00080000:3.0:1781049065.346674:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049065.346747:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049065.346756:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049065.346773:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513366592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781049065.346838:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366592:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781049065.346917:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049065.346921:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049065.348019:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513366528:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049065.348032:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513366464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049065.349293:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049065.349299:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049065.349306:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366592:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049065.985322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049066.241254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049067.009332:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049067.265324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049068.035036:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049068.289232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049069.058915:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049069.315787:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049070.081414:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049070.338273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049070.465430:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049070.465438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049070.465442:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049070.465454:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2706 00000100:00080000:1.0:1781049070.465458:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049070.465465:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049070.465513:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513366656:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781049070.465557:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049070.465581:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a5a8ed39 x1867562513366784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781049070.465682:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049070.465723:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049070.465735:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513366720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049070.465798:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049070.465803:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:3.0:1781049070.469569:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049070.469606:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513366656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049070.469772:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049070.469778:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049070.469954:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513366720:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049071.105591:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049071.361770:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049072.129293:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049072.385261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049073.153339:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049073.410785:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049074.177294:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049074.433262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049075.201365:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049075.457989:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781049075.585426:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513366848:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049075.585697:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049075.585706:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513366912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049075.586030:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049075.586036:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049075.586040:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049075.586046:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2711 00000100:00080000:0.0:1781049075.586064:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049075.586071:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049075.586082:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049075.586101:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513366976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049075.586121:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781049075.586153:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366976:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781049075.586156:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1781049075.587490:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049075.587496:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049075.587513:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513366976:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049075.587537:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513366848:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049075.587563:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513366912:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781049076.225380:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049076.481248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049077.249359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049077.505246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049078.273354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049078.529236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049079.297802:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049079.553251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049080.321337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049080.577211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049080.705478:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049080.705488:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049080.705491:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049080.705502:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2716 00000100:00080000:0.0:1781049080.705505:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049080.705512:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049080.705530:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049080.705535:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513367104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049080.705549:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513367168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049080.705576:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049080.705580:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049080.705626:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367168:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049080.705636:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781049080.705646:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513367040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049080.706928:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513367040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049080.706946:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513367104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049080.707814:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049080.707821:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049080.707827:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367168:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049081.345323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049081.601238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049082.369288:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049082.625260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049083.393320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049083.649238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049084.417302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049084.673242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049085.441354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049085.697241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049085.825352:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049085.825361:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049085.825365:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049085.825366:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513367296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049085.825375:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2721 00000100:00080000:0.0:1781049085.825379:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049085.825405:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049085.825416:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049085.825447:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513367360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049085.825471:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049085.825474:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049085.825492:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513367232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049085.825521:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049085.827112:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513367296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049085.827152:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513367232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049085.827212:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049085.827218:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049085.827223:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367360:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049086.465336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049086.721273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049087.489328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049087.745264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049088.513326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049088.769245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049089.537329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049089.793258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049090.561363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049090.817249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049090.945395:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049090.945416:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049090.945420:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049090.945424:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513367424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049090.945429:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2726 00000100:00080000:1.0:1781049090.945434:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049090.945441:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049090.945459:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049090.945476:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513367552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049090.945497:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049090.945501:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049090.945522:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513367488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049090.945544:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367552:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049090.947812:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513367424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049090.947812:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513367488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049090.947869:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049090.947877:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049090.947891:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367552:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049091.585299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049091.841262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049092.609344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049092.866416:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049093.633572:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049093.889258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049094.659392:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049094.913277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049095.681413:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049095.937219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049096.066245:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049096.066254:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049096.066257:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049096.066268:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2731 00000100:00080000:3.0:1781049096.066271:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049096.066292:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049096.066301:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049096.066315:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513367744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049096.066341:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049096.066345:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049096.066385:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367744:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049096.066434:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513367616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049096.066490:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049096.066497:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513367680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049096.067795:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513367680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049096.068349:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513367616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049096.068867:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049096.068873:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049096.068878:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367744:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049096.705377:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049096.963339:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049097.729305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049097.985199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049098.753541:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049099.009228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049099.777532:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049100.033279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049100.801338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049101.057261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049101.185285:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049101.185293:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049101.185297:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049101.185308:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2737 00000100:00100000:2.0:1781049101.185309:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513367808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049101.185312:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049101.185319:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049101.185329:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049101.185346:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513367936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049101.185365:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049101.185369:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049101.185386:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513367872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049101.185395:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367936:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049101.187370:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513367808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049101.187387:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513367872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049101.187417:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049101.187429:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049101.187434:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513367936:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049101.825397:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049102.081262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049102.850072:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049103.105232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049103.873341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049104.129252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049104.897346:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049105.154208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049105.922083:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049106.177235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049106.305272:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049106.305292:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049106.305295:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049106.305304:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2742 00000100:00080000:3.0:1781049106.305306:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049106.305313:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513368000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049106.305314:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049106.305322:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049106.305339:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513368128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049106.305360:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049106.305363:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049106.305383:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513368064:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049106.305410:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368128:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781049106.307096:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049106.307105:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049106.307114:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368128:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049106.307174:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513368000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049106.307230:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513368064:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049106.945471:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049107.201294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049107.969352:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049108.225274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049108.993287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049109.249291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049110.017409:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049110.273315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049111.041519:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049111.297941:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049111.425446:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049111.425454:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049111.425458:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049111.425470:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2747 00000100:00080000:0.0:1781049111.425474:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049111.425481:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049111.425492:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049111.425508:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513368320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049111.425526:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049111.425529:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049111.425586:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368320:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049111.425603:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513368192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049111.425699:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049111.425708:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513368256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049111.427057:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513368256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049111.427078:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513368192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049111.427106:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049111.427117:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049111.427156:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368320:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049112.065378:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049112.324089:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049113.089325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049113.345224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049114.113353:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049114.369254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049115.139103:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049115.393257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049116.161388:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049116.417262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049116.545523:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049116.545531:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049116.545534:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049116.545545:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2752 00000100:00080000:2.0:1781049116.545548:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049116.545555:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049116.545566:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049116.545582:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513368512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781049116.545586:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513368384:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049116.545637:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368512:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781049116.545692:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781049116.545692:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:2.0:1781049116.545697:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049116.545700:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513368448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049116.550625:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513368448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049116.550663:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513368384:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049116.553932:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049116.553943:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049116.553953:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368512:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049117.185321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049117.441254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049118.210076:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049118.465258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049119.234066:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049119.489269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049120.257329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049120.514372:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049121.283103:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049121.537244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049121.665524:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049121.665533:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049121.665537:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049121.665548:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2757 00000100:00080000:0.0:1781049121.665568:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049121.665575:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049121.665586:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049121.665605:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513368704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049121.665631:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049121.665635:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049121.665725:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513368640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049121.665779:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368704:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049121.665935:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781049121.665945:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513368576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049121.677327:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513368576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049121.677355:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513368640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049121.680461:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049121.680474:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049121.680484:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368704:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049122.305349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049122.561277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049123.329509:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049123.585272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049124.353986:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049124.609280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049125.377794:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049125.633842:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049126.401363:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049126.657282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049126.785343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049126.785350:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049126.785367:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049126.785377:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2762 00000100:00080000:1.0:1781049126.785379:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781049126.785388:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513368768:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049126.785394:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049126.785407:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049126.785426:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001d23a43b x1867562513368896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049126.785449:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049126.785453:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049126.785471:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513368832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049126.785496:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368896:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049126.786796:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513368768:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049126.786802:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513368832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049126.787208:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049126.787212:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049126.787217:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513368896:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049127.426254:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049127.681287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049128.449360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049128.705253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049129.473386:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049129.729416:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049130.497481:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049130.753262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049131.524362:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049131.777273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049131.905293:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049131.905301:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049131.905306:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049131.905317:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2767 00000100:00080000:1.0:1781049131.905322:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781049131.905329:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513368960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049131.905330:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049131.905344:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049131.905362:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513369088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049131.905400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049131.905404:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049131.905438:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049131.905446:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513369024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049131.905456:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369088:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049131.907912:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513369024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049131.907947:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513368960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049131.907955:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049131.907964:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049131.907973:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369088:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049132.545458:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049132.801403:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049133.569960:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049133.827725:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049134.594028:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049134.849277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049135.617415:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049135.873277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049136.641315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049136.897255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049137.025901:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049137.025909:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049137.025912:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049137.025922:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2772 00000100:00080000:0.0:1781049137.025925:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049137.025931:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049137.025941:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049137.025970:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513369280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049137.025994:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049137.025997:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049137.026021:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513369216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049137.026116:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781049137.026122:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513369152:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049137.026290:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369280:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049137.029087:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513369152:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049137.029116:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513369216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049137.029264:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049137.029269:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049137.029272:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369280:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049137.665286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049137.922618:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049138.689334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049138.945689:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049139.717088:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049139.969208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049140.737513:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049140.993628:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049141.761581:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049142.017338:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781049142.145662:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513369344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049142.147611:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049142.147631:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513369408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049142.149099:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049142.149110:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049142.149114:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049142.149141:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2778 00000100:00080000:2.0:1781049142.149145:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049142.149152:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049142.149165:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049142.149417:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c91281b8 x1867562513369472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049142.149452:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049142.149456:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049142.149891:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369472:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049142.150357:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513369408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049142.150912:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513369344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049142.151428:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049142.151434:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049142.151438:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369472:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049142.785329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049143.041304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049143.810993:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049144.065493:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049144.837883:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049145.089264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049145.857376:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049146.113236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049146.881283:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049147.137586:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049147.269672:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049147.269681:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049147.269685:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049147.269696:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2783 00000100:00100000:2.0:1781049147.269697:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513369536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049147.269700:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049147.269710:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049147.269722:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049147.269797:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049147.269805:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513369600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049147.269810:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513369664/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049147.269834:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049147.269837:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049147.269878:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369664:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1781049147.272200:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049147.272210:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049147.272236:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369664:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049147.273480:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513369600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049147.273510:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513369536:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049147.905627:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049148.162024:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049148.929306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049149.185446:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049149.955648:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049150.209263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049150.977514:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049151.233219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049152.001441:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049152.257300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049152.386566:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049152.386574:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049152.386579:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049152.386593:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2788 00000100:00080000:3.0:1781049152.386597:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049152.386607:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513369728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049152.386621:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049152.386634:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049152.386654:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000288f7157 x1867562513369856/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049152.386683:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049152.386687:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049152.386715:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513369792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049152.390169:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513369728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049152.390456:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369856:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049152.390603:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513369792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049152.391867:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049152.391875:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049152.391881:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513369856:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049153.025416:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049153.281293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049154.049359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049154.305297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049155.073451:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049155.329461:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049156.097405:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049156.353287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049157.121808:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049157.378731:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049157.505373:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049157.505382:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049157.505386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049157.505398:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2793 00000100:00080000:0.0:1781049157.505402:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781049157.505406:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513369920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049157.505411:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049157.505425:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049157.505443:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513370048/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049157.505476:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049157.505480:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049157.505503:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513369984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049157.505622:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370048:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049157.508552:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513369920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049157.509601:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049157.509611:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049157.509618:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370048:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049157.511205:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513369984:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049158.145375:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049158.401745:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049159.170621:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049159.425322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049160.197112:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049160.449287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049161.217313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049161.473243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049162.241378:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049162.497344:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781049162.625434:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513370112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049162.625578:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049162.625586:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513370176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049162.625663:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049162.625671:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049162.625676:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049162.625681:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2798 00000100:00080000:1.0:1781049162.625684:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049162.625689:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049162.625700:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049162.625741:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f819707f x1867562513370240/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049162.625765:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049162.625769:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049162.625812:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370240:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781049162.627725:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049162.627736:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049162.627744:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370240:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049162.627840:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513370176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049162.628166:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513370112:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049163.265389:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049163.521482:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049164.289339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049164.545310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049165.313380:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049165.569371:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049166.337371:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049166.593290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049167.361833:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049167.617684:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049167.745355:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049167.745362:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049167.745366:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049167.745377:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2803 00000100:00080000:1.0:1781049167.745380:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049167.745385:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049167.745395:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049167.745407:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513370432/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049167.745425:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049167.745428:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049167.745460:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513370368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049167.745520:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370432:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049167.745555:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781049167.745564:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513370304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049167.747910:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513370368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049167.748625:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513370304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049167.748642:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049167.748653:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049167.748662:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370432:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049168.385370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049168.641310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049169.409342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049169.665352:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049170.433400:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049170.689225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049171.457399:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049171.713310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049172.481368:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049172.737214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049172.865303:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049172.865312:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049172.865316:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049172.865327:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2808 00000100:00080000:3.0:1781049172.865332:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049172.865339:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781049172.865344:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513370496:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781049172.865350:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049172.865370:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513370624/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049172.865413:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049172.865418:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049172.865439:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049172.865448:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513370560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049172.865461:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370624:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:02020000:1.0:1781049172.867456:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:1.0:1781049172.867467:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049172.867470:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049172.867472:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370624:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049172.869978:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513370560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049172.869996:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513370496:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049173.505356:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049173.761275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049174.530038:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049174.785299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049175.553430:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049175.809260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049176.577465:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049176.833257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049177.601564:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049177.857262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049177.985918:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049177.985927:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049177.985930:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049177.985940:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2813 00000100:00080000:0.0:1781049177.985944:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049177.985949:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049177.985959:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049177.985975:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513370816/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049177.985996:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049177.986000:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049177.986034:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513370752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049177.986042:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370816:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049177.986103:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513370688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049177.989448:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513370752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049177.989512:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049177.989517:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049177.989532:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513370816:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049177.990438:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513370688:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049178.625906:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049178.882464:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049179.649434:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049179.905264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049180.673321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049180.929270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049181.697302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049181.953248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049182.721412:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049182.977259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781049183.106972:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513370880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049183.107383:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049183.107388:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049183.107390:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049183.107395:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2819 00000100:00080000:1.0:1781049183.107398:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049183.107404:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049183.107412:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513370944:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781049183.107413:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049183.107426:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513371008/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049183.107441:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049183.107451:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049183.107488:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049183.109973:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513370880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049183.110077:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049183.110082:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049183.110086:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049183.110440:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513370944:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781049183.745428:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049184.001275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049184.769360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049185.025362:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049185.793335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049186.049265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049186.817419:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049187.073337:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049187.841269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049188.098191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049188.225338:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049188.225348:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049188.225352:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049188.225364:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2824 00000100:00080000:1.0:1781049188.225368:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049188.225385:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049188.225397:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049188.225413:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513371200/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049188.225433:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049188.225437:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049188.225487:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371200:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049188.225609:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513371136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049188.225654:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781049188.225661:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513371072:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049188.227351:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049188.227358:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513371072:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049188.227362:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049188.227371:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371200:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049188.227383:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513371136:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049188.865396:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049189.121262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049189.889660:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049190.145258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049190.913357:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049191.169272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049191.937444:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049192.193275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049192.961335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049193.217283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049193.345388:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049193.345397:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049193.345400:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049193.345412:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2829 00000100:00100000:2.0:1781049193.345413:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513371264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049193.345415:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049193.345420:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049193.345432:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049193.345448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513371392/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049193.345483:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049193.345486:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049193.345509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049193.345535:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513371328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049193.349559:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513371328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049193.349614:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513371264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049193.349637:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049193.349643:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049193.349649:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371392:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049193.985649:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049194.241250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049195.009890:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049195.265331:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049196.033378:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049196.289300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049197.057296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049197.313252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049198.081353:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049198.337266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049198.465343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049198.465360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049198.465365:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049198.465389:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2834 00000100:00080000:3.0:1781049198.465393:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049198.465399:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049198.465411:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049198.465416:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513371456:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049198.465432:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513371584/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049198.465458:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049198.465462:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049198.465491:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513371520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049198.465503:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371584:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049198.468012:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513371456:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049198.468058:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049198.468063:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049198.468068:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371584:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049198.468094:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513371520:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049199.105372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049199.361265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049200.129384:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049200.385500:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049201.153366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049201.409281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049202.177366:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049202.433402:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049203.201269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049203.457183:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049203.585393:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049203.585413:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049203.585416:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049203.585427:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2839 00000100:00100000:2.0:1781049203.585430:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513371648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049203.585431:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049203.585440:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049203.585453:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049203.585473:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513371776/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049203.585493:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049203.585496:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049203.585522:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513371712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049203.585560:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049203.587775:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513371712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049203.587777:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049203.587784:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049203.587791:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049203.588101:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513371648:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049204.225348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049204.481240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049205.249301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049205.505277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049206.273322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049206.529268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049207.297317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049207.553243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049208.321355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049208.577293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049208.705323:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049208.705332:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049208.705335:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049208.705347:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2844 00000100:00080000:0.0:1781049208.705350:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781049208.705357:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513371840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049208.705358:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049208.705371:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049208.705387:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513371968/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049208.705410:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049208.705413:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049208.705434:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513371904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049208.705459:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371968:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049208.706694:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513371840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049208.706749:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513371904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049208.706757:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049208.706761:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049208.706769:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513371968:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049209.345373:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049209.601271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049210.369374:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049210.625290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049211.393374:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049211.649257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049212.417350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049212.673247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049213.441241:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049213.697267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049213.825354:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049213.825361:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049213.825366:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049213.825378:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049213.825378:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2849 00000100:00080000:3.0:1781049213.825384:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049213.825391:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049213.825418:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049213.825436:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513372160/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049213.825462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049213.825466:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049213.825469:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049213.825478:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513372096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049213.825510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372160:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049213.826548:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049213.826667:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513372096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049213.826673:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049213.826679:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049213.826684:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372160:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049214.465347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049214.721264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049215.489347:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049215.745266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049216.513315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049216.769261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049217.537343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049217.793281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049218.561332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049218.817230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049218.945344:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049218.945353:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049218.945358:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049218.945371:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2854 00000100:00080000:3.0:1781049218.945374:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049218.945378:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049218.945384:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049218.945400:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049218.945418:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513372352/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049218.945436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049218.945439:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049218.945460:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513372288:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049218.945488:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372352:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049218.946565:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049218.946675:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513372288:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049218.946729:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049218.946736:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049218.946742:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372352:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049219.585343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049219.841245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049220.609290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049220.865220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049221.634094:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049221.889247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049222.657304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049222.913214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049223.681328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049223.937271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049224.065304:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049224.065312:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049224.065315:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049224.065329:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2859 00000100:00080000:3.0:1781049224.065333:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049224.065340:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781049224.065377:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372416:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781049224.065412:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049224.065436:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513372544/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049224.065459:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049224.065463:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049224.065489:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513372480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049224.065508:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372544:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049224.066631:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372416:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049224.066653:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513372480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049224.066692:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049224.066697:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049224.066702:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372544:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049224.705258:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049224.961254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049225.729337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049225.985264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049226.753337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049227.009201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049227.777343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049228.033342:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049228.801395:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049229.057266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049229.185329:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049229.185337:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049229.185340:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049229.185349:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513372608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049229.185349:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2865 00000100:00080000:1.0:1781049229.185352:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049229.185357:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049229.185368:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049229.185382:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c26c4b8c x1867562513372736/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049229.185405:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049229.185408:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049229.185440:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513372672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049229.185445:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372736:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049229.186988:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513372672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049229.186994:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049229.186999:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049229.187004:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372736:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049229.187038:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513372608:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049229.825317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049230.081274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049230.849337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049231.105270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049231.873351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049232.129253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049232.899112:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049233.153253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049233.921330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049234.177273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049234.305352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049234.305358:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049234.305364:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049234.305376:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2870 00000100:00080000:3.0:1781049234.305394:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049234.305397:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049234.305404:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049234.305418:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049234.305438:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513372928/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049234.305461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049234.305466:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049234.305489:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513372864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049234.305515:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372928:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049234.307246:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513372864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049234.307299:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049234.307317:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049234.307323:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049234.307328:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513372928:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049234.945318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049235.201236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049235.969296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049236.225225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049236.993304:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049237.249236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049238.017916:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049238.273224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049239.041283:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049239.297234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049239.425278:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049239.425284:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049239.425287:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049239.425295:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2875 00000100:00080000:3.0:1781049239.425299:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049239.425304:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049239.425313:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049239.425317:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372992:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049239.425327:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513373120/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049239.425343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049239.425345:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049239.425376:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513373056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049239.425387:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373120:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049239.426710:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513372992:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049239.426725:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513373056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049239.426744:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049239.426750:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049239.426755:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373120:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049240.065379:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049240.321217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049241.089330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049241.345267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049242.113264:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049242.369285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049243.137351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049243.393270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049244.161320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049244.417280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049244.545288:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049244.545296:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049244.545300:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049244.545311:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2880 00000100:00080000:3.0:1781049244.545315:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049244.545324:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049244.545335:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049244.545355:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513373312/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781049244.545362:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513373184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049244.545385:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049244.545396:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049244.545427:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513373248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049244.545428:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373312:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049244.546853:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513373248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049244.546897:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513373184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049244.546912:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049244.546918:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049244.546936:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373312:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049245.185308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049245.441244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049246.209319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049246.465194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049247.233319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049247.489230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049248.257258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049248.513223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049249.281299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049249.537257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049249.665282:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049249.665290:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049249.665294:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049249.665305:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2885 00000100:00100000:2.0:1781049249.665315:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513373376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049249.665325:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049249.665340:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049249.665348:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049249.665364:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513373504/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049249.665380:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049249.665383:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049249.665416:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049249.665426:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513373440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049249.665436:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373504:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049249.668895:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513373440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049249.668918:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513373376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049249.668940:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049249.668948:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049249.668953:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373504:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049250.305323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049250.561260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049251.329322:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049251.585289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049252.353317:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049252.609264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049253.377365:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049253.633248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049254.401305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049254.657253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049254.785371:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049254.785379:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049254.785382:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049254.785393:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2890 00000100:00080000:1.0:1781049254.785397:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 6s 00000100:00080000:1.0:1781049254.785404:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049254.785415:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049254.785429:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b98b0b8e x1867562513373696/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049254.785465:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049254.785467:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049254.785490:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513373632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049254.785511:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373696:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049254.785641:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513373568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049254.787036:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513373632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049254.787043:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049254.787048:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049254.787054:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373696:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049254.789193:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513373568:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049255.425286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049255.681265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049256.449341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049256.705237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049257.473293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049257.729215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049258.497336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049258.753231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049259.521386:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049259.777185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049259.905321:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049259.905327:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049259.905330:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049259.905339:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2895 00000100:00080000:0.0:1781049259.905343:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 11s 00000100:00080000:0.0:1781049259.905349:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049259.905357:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049259.905367:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513373760:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049259.905371:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513373888/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049259.905389:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049259.905392:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049259.905422:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513373824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049259.905443:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373888:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049259.906864:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513373760:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049259.906907:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513373824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049259.906911:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049259.906919:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049259.906927:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513373888:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049260.545327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049260.801297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049261.569291:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049261.825290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049262.593329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049262.849270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049263.617280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049263.873295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049264.641363:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049264.897270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049265.025409:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049265.025418:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049265.025422:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049265.025445:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513373952:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049265.025448:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2900 00000100:00080000:0.0:1781049265.025455:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00080000:0.0:1781049265.025462:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049265.025474:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049265.025493:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513374080/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049265.025518:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049265.025522:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049265.025546:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049265.025561:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374080:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049265.026668:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513373952:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049265.026699:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049265.026975:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049265.026980:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049265.026985:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374080:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049265.665363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049265.921243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049266.689336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049266.945265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049267.713329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049267.971354:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049268.737303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049268.993271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049269.761290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049270.017286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049270.145322:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049270.145328:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049270.145332:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049270.145343:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2906 00000100:00080000:0.0:1781049270.145347:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049270.145352:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781049270.145360:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374144:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781049270.145362:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049270.145380:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513374272/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049270.145394:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049270.145396:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049270.145411:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374208:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049270.145432:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374272:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049270.146714:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049270.146760:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374208:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049270.146768:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049270.146772:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049270.146776:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374272:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049270.785269:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049271.041241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049271.809306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049272.065249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049272.833239:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049273.089227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049273.857316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049274.113276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049274.881319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049275.137229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049275.265309:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049275.265315:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049275.265318:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049275.265328:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2911 00000100:00080000:0.0:1781049275.265331:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049275.265336:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049275.265345:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049275.265351:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049275.265359:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513374464/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049275.265375:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049275.265378:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049275.265397:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374400:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049275.265414:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374464:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049275.266862:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374400:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049275.266909:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049275.266936:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049275.266941:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049275.266945:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374464:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049275.905333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049276.161295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049276.929327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049277.185200:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049277.953320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049278.209258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049278.977286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049279.233274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049280.001329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049280.257273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049280.385304:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049280.385312:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049280.385316:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049280.385318:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049280.385339:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2916 00000100:00080000:1.0:1781049280.385343:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049280.385349:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049280.385362:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049280.385395:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001cbc13b4 x1867562513374656/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049280.385416:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049280.385420:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049280.385443:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049280.385461:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374656:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049280.386633:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049280.386633:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049280.386654:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049280.386659:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049280.386666:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374656:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049281.025277:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049281.281247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049282.049325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049282.305243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049283.073321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049283.329226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049284.097319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049284.353245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049285.121315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049285.377226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049285.505376:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049285.505384:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049285.505387:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049285.505396:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049285.505400:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2921 00000100:00080000:0.0:1781049285.505407:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049285.505415:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049285.505443:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049285.505464:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513374848/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049285.505494:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049285.505498:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049285.505501:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049285.505510:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513374784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049285.505544:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374848:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049285.506652:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513374784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049285.506661:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781049285.506666:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049285.506668:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049285.506688:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513374848:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049286.145284:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049286.401241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049287.169301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049287.425271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049288.193263:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049288.449239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049289.217292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049289.473263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049290.241294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049290.497246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049290.625302:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049290.625311:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049290.625314:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049290.625324:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2926 00000100:00080000:0.0:1781049290.625328:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049290.625336:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049290.625347:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049290.625364:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513375040/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781049290.625371:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049290.625392:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049290.625397:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049290.625422:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049290.625450:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375040:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049290.626684:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513374976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049290.626736:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513374912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049290.626767:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049290.626771:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049290.626776:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375040:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049291.265320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049291.521247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049292.289309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049292.545225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049293.313308:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049293.569250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049294.337319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049294.593203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049295.361354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049295.617252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049295.745331:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049295.745337:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049295.745341:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049295.745351:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2931 00000100:00100000:0.0:1781049295.745355:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049295.745355:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049295.745362:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049295.745371:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049295.745385:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513375232/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049295.745402:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049295.745404:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049295.745452:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375232:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049295.745461:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781049295.745470:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513375104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049295.746696:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513375104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049295.746699:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049295.746704:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049295.746708:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375232:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049295.746713:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375168:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049296.385288:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049296.641188:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049297.409294:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049297.665240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049298.433251:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049298.689211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049299.457283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049299.713225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049300.481238:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049300.737239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049300.865331:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049300.865339:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781049300.865340:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049300.865342:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049300.865346:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2936 00000100:00080000:0.0:1781049300.865349:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049300.865355:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049300.865365:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049300.865386:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513375424/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049300.865405:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049300.865408:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049300.865433:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049300.865442:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375424:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049300.866638:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049300.866645:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049300.866705:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049300.866709:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049300.866713:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375424:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049301.505287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049301.761241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049302.529245:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049302.785187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049303.553290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049303.809221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049304.577281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049304.833215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049305.601277:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049305.857237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049305.985314:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049305.985323:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049305.985327:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049305.985338:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2941 00000100:00080000:0.0:1781049305.985343:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049305.985350:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781049305.985353:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375488:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781049305.985378:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049305.985396:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513375616/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049305.985427:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049305.985431:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049305.985450:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049305.985463:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375616:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049305.987530:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049305.987564:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049305.987570:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049305.987574:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375616:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049305.988498:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375552:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781049306.625305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049306.881237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049307.649304:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049307.905213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049308.673381:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049308.929253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049309.697269:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049309.953224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049310.721320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049310.977271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049311.105434:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049311.105440:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049311.105443:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049311.105453:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2947 00000100:00100000:0.0:1781049311.105468:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049311.105471:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049311.105479:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049311.105490:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049311.105511:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513375808/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049311.105527:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049311.105530:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049311.105577:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375808:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049311.105583:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049311.106824:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049311.106840:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049311.106846:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049311.106851:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513375808:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049311.106890:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375680:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049311.745287:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049312.001246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049312.769287:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049313.025285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049313.793322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049314.049204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049314.817344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049315.073279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049315.841367:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049316.097188:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049316.225346:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049316.225355:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049316.225359:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049316.225364:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049316.225368:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2952 00000100:00080000:0.0:1781049316.225374:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049316.225380:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049316.225390:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049316.225406:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513376000/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049316.225429:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049316.225433:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049316.225455:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049316.225475:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376000:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049316.226731:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513375872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049316.226750:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513375936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049316.226810:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049316.226815:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049316.226819:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376000:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049316.865323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049317.121252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049317.889297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049318.145241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049318.913308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049319.169244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049319.937267:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049320.193232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049320.961314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049321.217252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049321.345334:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049321.345342:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049321.345346:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049321.345358:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2957 00000100:00080000:0.0:1781049321.345361:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781049321.345364:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049321.345371:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049321.345381:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049321.345399:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513376192/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049321.345421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049321.345425:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049321.345448:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049321.345460:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376192:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049321.346646:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049321.346674:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049321.346677:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049321.346682:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049321.346688:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376192:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049321.985303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049322.241256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049323.009245:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049323.265256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049324.033312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049324.289197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049325.057306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049325.313264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049326.081313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049326.337251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049326.465339:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049326.465346:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049326.465350:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049326.465361:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2962 00000100:00100000:0.0:1781049326.465362:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049326.465365:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049326.465372:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049326.465382:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049326.465396:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513376384/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049326.465411:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049326.465413:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049326.465448:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049326.465464:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376384:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049326.466813:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049326.466815:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:2.0:1781049326.466832:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:2.0:1781049326.472308:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049326.472314:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049326.472324:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376384:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049327.105303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049327.361226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049328.129282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049328.385253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049329.153273:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049329.409262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049330.177310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049330.433248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049331.201358:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049331.457244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049331.585321:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049331.585327:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049331.585330:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049331.585334:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049331.585339:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2967 00000100:00080000:0.0:1781049331.585360:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049331.585369:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049331.585379:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049331.585397:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018617087 x1867562513376576/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049331.585420:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1781049331.585420:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:0.0:1781049331.585425:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049331.585426:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513376512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049331.585461:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376576:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049331.586556:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513376512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049331.586572:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781049331.586574:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049331.586578:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049331.586584:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376576:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049332.225303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049332.481249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049333.249266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049333.505225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049334.273280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049334.529233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049335.297261:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049335.553193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049336.321259:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049336.577256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049336.705292:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049336.705301:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049336.705305:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049336.705316:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2972 00000100:00080000:0.0:1781049336.705320:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049336.705327:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049336.705339:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049336.705343:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376640:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049336.705356:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513376768/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049336.705382:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049336.705386:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049336.705407:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376704:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049336.705433:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376768:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049336.707012:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376704:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049336.707077:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049336.707096:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049336.707101:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376768:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049336.707224:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376640:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049337.345260:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049337.601228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049338.369296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049338.625221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049339.393278:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049339.649233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049340.417265:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049340.673223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049341.441314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049341.697216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049341.825317:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049341.825325:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049341.825343:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049341.825351:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2977 00000100:00080000:1.0:1781049341.825354:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049341.825359:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049341.825367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049341.825379:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ce8e0ca x1867562513376960/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049341.825397:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781049341.825400:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376896:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049341.825400:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049341.825433:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049341.825439:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376960:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049341.826695:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513376832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049341.826709:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513376896:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049341.826725:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049341.826729:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049341.826734:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513376960:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049342.465307:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049342.721263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049343.489258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049343.745225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049344.513270:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049344.769240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049345.537263:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049345.793196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049346.561285:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049346.817254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049346.945284:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049346.945293:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781049346.945296:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049346.945297:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049346.945306:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2982 00000100:00080000:0.0:1781049346.945309:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049346.945316:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049346.945325:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049346.945354:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513377152/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049346.945375:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049346.945378:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049346.945397:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377088:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049346.945413:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377152:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049346.946569:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377088:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049346.946569:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781049346.946570:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049346.946575:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049346.946584:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377152:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049347.585280:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049347.841244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049348.609305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049348.865220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049349.633260:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049349.889225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049350.657266:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049350.913251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049351.681314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049351.937251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049352.065304:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049352.065310:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049352.065312:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049352.065321:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2987 00000100:00080000:1.0:1781049352.065324:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1781049352.065328:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049352.065332:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049352.065342:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049352.065357:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003ce8e0ca x1867562513377344/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049352.065371:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049352.065373:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049352.065406:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049352.065417:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377344:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049352.066630:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049352.066631:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049352.066680:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049352.066685:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049352.066690:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377344:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049352.705320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049352.961212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049353.729322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049353.985258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049354.753290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049355.009186:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049355.777274:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049356.033231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049356.801314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049357.057219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049357.185340:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049357.185348:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049357.185351:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049357.185374:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2993 00000100:00100000:1.0:1781049357.185374:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049357.185393:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049357.185398:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049357.185409:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049357.185426:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513377536/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049357.185444:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049357.185447:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049357.185477:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049357.185509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377536:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049357.186589:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049357.186627:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049357.186715:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049357.186719:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049357.186724:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377536:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049357.825310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049358.081221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049358.849318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049359.105233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049359.873333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049360.129220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049360.897288:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049361.153187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049361.921318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049362.177225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049362.305279:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049362.305286:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049362.305289:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049362.305298:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 2998 00000100:00080000:0.0:1781049362.305302:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781049362.305302:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049362.305309:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049362.305320:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049362.305334:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513377728/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049362.305351:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049362.305354:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049362.305371:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049362.305420:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377728:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049362.306646:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049362.306691:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049362.306707:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049362.306711:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049362.306715:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377728:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049362.945332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049363.201201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049363.969286:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049364.225210:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049364.993327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049365.249261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049366.017283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049366.273209:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049367.041330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049367.297235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049367.425340:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049367.425349:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049367.425353:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049367.425356:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049367.425363:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3003 00000100:00080000:0.0:1781049367.425367:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049367.425374:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049367.425386:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049367.425407:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513377920/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049367.425431:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049367.425436:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049367.425461:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049367.425474:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377920:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049367.426724:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049367.426753:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049367.426755:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513377856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049367.426759:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049367.426766:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513377920:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049368.065269:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049368.321206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049369.089358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049369.345229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049370.113358:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049370.369241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049371.137355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049371.393292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049372.161345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049372.417196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049372.545318:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049372.545325:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049372.545328:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049372.545338:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3008 00000100:00080000:0.0:1781049372.545341:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049372.545347:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049372.545356:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049372.545372:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049372.545396:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c99a9336 x1867562513378112/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049372.545422:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049372.545425:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049372.545445:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513378048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049372.545514:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378112:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049372.546790:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513377984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049372.546902:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513378048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049372.546907:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049372.546913:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049372.546919:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378112:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049373.185347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049373.441285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049374.209375:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049374.465240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049375.233356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049375.489204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049376.257306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049376.513283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049377.281354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049377.537267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049377.665373:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049377.665380:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049377.665383:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049377.665395:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3013 00000100:00080000:1.0:1781049377.665398:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049377.665404:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049377.665415:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049377.665419:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513378240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049377.665430:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513378304/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049377.665447:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049377.665450:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049377.665488:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513378176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049377.665505:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378304:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049377.666885:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513378240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049377.666922:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513378176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049377.666950:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049377.666955:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049377.666960:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378304:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049378.305273:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049378.561255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049379.329309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049379.585276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049380.353336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049380.609244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049381.377341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049381.633278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049382.401326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049382.657205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049382.785272:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049382.785279:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049382.785283:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049382.785294:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3018 00000100:00080000:1.0:1781049382.785298:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049382.785304:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049382.785311:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513378368:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781049382.785314:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049382.785336:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513378496/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049382.785352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049382.785355:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049382.785391:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513378432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049382.785403:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378496:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049382.786722:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513378368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049382.786783:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513378432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049382.786834:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049382.786839:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049382.786845:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378496:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049383.425263:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049383.681220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049384.449323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049384.705231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049385.473259:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049385.729248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049386.497300:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049386.753272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049387.521281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049387.777236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049387.905444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049387.905451:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049387.905455:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049387.905467:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3023 00000100:00100000:0.0:1781049387.905478:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513378560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049387.905485:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049387.905492:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049387.905503:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049387.905518:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513378688/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781049387.905588:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1781049387.905593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781049387.905598:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513378624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049387.905599:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049387.905641:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378688:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049387.907096:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513378624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049387.907118:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049387.907151:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049387.907158:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378688:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049387.907166:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513378560:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049388.545312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049388.801269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049389.569304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049389.825271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049390.593303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049390.849244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049391.617317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049391.873210:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049392.641318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049392.897242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049393.025325:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049393.025356:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049393.025360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049393.025373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3028 00000100:00100000:3.0:1781049393.025373:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513378752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049393.025378:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049393.025388:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049393.025399:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049393.025418:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513378880/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049393.025439:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049393.025443:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049393.025469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378880:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049393.025474:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049393.025485:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513378816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049393.026703:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513378816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049393.026719:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513378752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049393.026745:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049393.026751:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049393.026758:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513378880:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049393.665263:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049393.921273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049394.689330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049394.945239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049395.713268:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049395.969219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049396.737331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049396.993259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049397.761297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049398.017219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049398.145372:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049398.145381:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049398.145386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049398.145388:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513378944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049398.145396:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3034 00000100:00080000:3.0:1781049398.145401:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781049398.145409:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049398.145421:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049398.145440:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513379072/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781049398.145476:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:3.0:1781049398.145478:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781049398.145483:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513379008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049398.145486:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049398.145534:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049398.147017:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513379008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049398.147032:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513378944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049398.147114:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049398.147118:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049398.147147:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379072:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049398.785309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049399.041232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049399.809296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049400.065245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049400.833316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049401.089253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049401.857346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049402.113297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049402.881325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049403.137249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049403.265244:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049403.265252:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049403.265256:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049403.265268:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3039 00000100:00080000:3.0:1781049403.265271:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049403.265278:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049403.265287:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049403.265291:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049403.265306:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513379264/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049403.265331:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049403.265334:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049403.265355:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049403.265377:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379264:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049403.266570:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049403.266570:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049403.266587:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049403.266594:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049403.266600:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379264:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049403.905326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049404.161237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049404.929236:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049405.185208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049405.953285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049406.209257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049406.977274:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049407.233228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049408.001287:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049408.257279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049408.385305:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049408.385312:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049408.385316:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049408.385326:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3044 00000100:00080000:3.0:1781049408.385343:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049408.385345:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379328:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049408.385350:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049408.385361:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049408.385375:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513379456/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049408.385390:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049408.385392:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049408.385410:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049408.385448:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379456:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049408.386725:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379328:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049408.386753:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049408.386773:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049408.386781:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049408.386785:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379456:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049409.025327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049409.281192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049410.049301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049410.305244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049411.073271:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049411.329193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049412.097328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049412.353198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049413.121305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049413.377212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049413.505260:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049413.505267:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049413.505270:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049413.505282:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3049 00000100:00080000:3.0:1781049413.505285:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049413.505291:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049413.505301:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049413.505307:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049413.505318:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513379648/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049413.505338:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049413.505342:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049413.505377:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049413.505382:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379648:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049413.506598:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049413.506626:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049413.506637:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049413.506641:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049413.506645:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379648:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049414.145352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049414.401198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049415.169323:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049415.425199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049416.193315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781049416.449256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049417.217329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049417.473249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049418.241307:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049418.497260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049418.625302:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049418.625307:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049418.625310:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049418.625320:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3054 00000100:00080000:3.0:1781049418.625322:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049418.625327:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049418.625337:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049418.625350:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049418.625354:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513379840/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049418.625382:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049418.625385:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049418.625416:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049418.625425:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379840:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049418.626555:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049418.626557:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049418.626567:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049418.626572:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049418.626578:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513379840:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049419.265348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049419.521204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049420.289313:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049420.545261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049421.313351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049421.569229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049422.337313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049422.593226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049423.361337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049423.617202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049423.745319:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049423.745340:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049423.745344:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049423.745352:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3059 00000100:00100000:2.0:1781049423.745353:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049423.745355:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049423.745363:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049423.745372:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049423.745386:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513380032/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049423.745402:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049423.745404:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049423.745421:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049423.745456:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380032:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049423.746592:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513379904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049423.746634:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513379968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049423.746709:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049423.746721:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049423.746725:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380032:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049424.385315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049424.641273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049425.409364:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049425.665270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049426.433278:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049426.689290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049427.457310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049427.713344:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049428.481303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049428.737242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049428.865343:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049428.865351:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049428.865355:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781049428.865370:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049428.865372:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3064 00000100:00080000:2.0:1781049428.865376:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049428.865382:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049428.865390:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049428.865406:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000015c9d1f3 x1867562513380224/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049428.865421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049428.865424:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049428.865460:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049428.865466:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513380160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049428.865471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380224:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049428.866725:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513380160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049428.866734:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049428.866739:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049428.866747:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380224:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049428.866753:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380096:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781049429.505260:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049429.761266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049430.529282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049430.785202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049431.553327:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049431.809250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049432.577267:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049432.833263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049433.601296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049433.857225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049433.985306:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781049433.985318:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049433.985329:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049433.985333:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049433.985339:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3069 00000100:00080000:3.0:1781049433.985342:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049433.985348:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049433.985358:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049433.985376:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513380416/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049433.985398:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049433.985400:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049433.985416:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049433.985425:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513380352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049433.985442:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1781049433.986649:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781049433.986651:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513380352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049433.986653:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049433.986657:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049433.986663:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380288:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049434.625320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049434.881225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049435.649289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049435.905239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049436.673288:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049436.929199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049437.697320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049437.953231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049438.721297:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049438.977208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049439.105300:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049439.105309:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049439.105313:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049439.105324:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049439.105325:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3075 00000100:00080000:3.0:1781049439.105330:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049439.105337:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049439.105346:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049439.105362:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513380608/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049439.105382:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049439.105385:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049439.105407:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049439.105415:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513380544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049439.105432:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380608:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049439.106597:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513380544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049439.106612:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049439.106639:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049439.106645:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049439.106650:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380608:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049439.745298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049440.001196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049440.769301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049441.025226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049441.793274:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049442.049239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049442.817355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049443.073277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049443.841286:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049444.097256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049444.225354:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049444.225362:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049444.225366:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049444.225376:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3080 00000100:00100000:3.0:1781049444.225378:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049444.225380:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049444.225389:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049444.225402:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049444.225418:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4afdb11 x1867562513380800/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049444.225452:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049444.225455:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049444.225479:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513380736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049444.225504:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380800:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049444.227034:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513380736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049444.227066:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049444.227083:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049444.227088:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049444.227097:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380800:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049444.865357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049445.121254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049445.889294:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049446.145227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049446.913345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049447.169262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049447.937330:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049448.193226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049448.961287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049449.217244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049449.345288:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049449.345296:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049449.345300:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049449.345314:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3085 00000100:00080000:2.0:1781049449.345317:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781049449.345323:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049449.345326:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049449.345339:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049449.345359:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4afdb11 x1867562513380992/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049449.345383:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049449.345387:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049449.345411:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513380928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049449.345417:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380992:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049449.346501:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513380928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049449.346540:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513380864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049449.346560:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049449.346567:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049449.346571:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513380992:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049449.985277:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049450.241261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049451.009271:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049451.265228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049452.033266:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049452.289235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049453.057285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049453.313220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049454.081239:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049454.337226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049454.465312:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049454.465320:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049454.465324:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049454.465335:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3090 00000100:00080000:3.0:1781049454.465337:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049454.465342:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049454.465344:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049454.465356:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049454.465373:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513381184/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049454.465408:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049454.465427:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049454.465439:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049454.465447:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513381120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049454.465453:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381184:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049454.466552:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513381120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049454.466565:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781049454.466570:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049454.466572:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049454.466578:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381184:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049455.105310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049455.361243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049456.129308:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049456.385224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049457.153312:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049457.409259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049458.177309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049458.433229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049459.201306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049459.457297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049459.585329:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049459.585338:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049459.585343:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049459.585364:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049459.585364:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3095 00000100:00080000:3.0:1781049459.585370:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049459.585378:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049459.585391:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049459.585409:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513381376/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049459.585429:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049459.585433:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049459.585462:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049459.585470:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381376:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049459.585470:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513381312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049459.588308:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781049459.588313:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513381312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049459.588315:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049459.588323:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381376:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049459.588335:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381248:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049460.225291:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049460.481235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049461.249265:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049461.505220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049462.273265:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049462.529249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049463.297267:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049463.553246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049464.321359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049464.577276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049464.705309:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049464.705315:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049464.705318:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049464.705328:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3100 00000100:00080000:0.0:1781049464.705331:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049464.705343:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781049464.705348:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513381440:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1781049464.705354:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049464.705369:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c99a9336 x1867562513381568/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049464.705387:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049464.705389:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049464.705420:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513381504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049464.705426:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381568:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049464.706676:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513381440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049464.706690:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513381504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049464.706698:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049464.706703:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049464.706708:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381568:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049465.345313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049465.601214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049466.369311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049466.625237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049467.393328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049467.649221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049468.417324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049468.673318:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049469.441346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049469.697262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049469.825369:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049469.825377:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049469.825381:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049469.825393:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3105 00000100:00100000:2.0:1781049469.825401:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049469.825412:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049469.825420:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049469.825445:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049469.825461:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513381760/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049469.825482:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049469.825485:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049469.825497:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049469.825506:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513381696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049469.825524:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381760:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049469.826698:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049469.826724:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513381696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049469.826757:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049469.826762:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049469.826766:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381760:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049470.465289:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049470.721254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049471.489294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049471.745212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049472.513283:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049472.769228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049473.537319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049473.793192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049474.561382:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049474.817270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049474.945284:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049474.945292:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049474.945294:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049474.945309:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3110 00000100:00080000:3.0:1781049474.945312:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049474.945325:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049474.945335:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049474.945367:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513381952/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781049474.945375:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049474.945400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049474.945403:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049474.945429:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513381888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049474.945471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381952:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049474.946938:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513381888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049474.947027:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513381824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049474.947034:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049474.947042:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049474.947049:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513381952:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049475.585307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049475.841247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049476.609326:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049476.865213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049477.633321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049477.889200:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049478.657317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049478.913230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049479.681369:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049479.937236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049480.065414:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049480.065422:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049480.065427:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049480.065441:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3115 00000100:00080000:2.0:1781049480.065446:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049480.065453:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049480.065466:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049480.065485:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000015c9d1f3 x1867562513382144/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049480.065513:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049480.065517:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049480.065555:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513382080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049480.065576:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382144:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049480.065608:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513382016:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049480.066845:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513382080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049480.066863:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513382016:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:1.0:1781049480.067023:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:1.0:1781049480.067029:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049480.067032:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049480.067035:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382144:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049480.705282:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049480.961253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049481.729356:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049481.985280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049482.753310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049483.009203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049483.777389:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049484.033308:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049484.801370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049485.057233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049485.185389:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049485.185412:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049485.185416:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049485.185427:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3121 00000100:00100000:0.0:1781049485.185431:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513382208:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049485.185431:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049485.185442:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049485.185457:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049485.185478:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513382336/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049485.185501:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049485.185504:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049485.185531:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049485.185539:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513382272:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049485.185549:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382336:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049485.187016:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513382272:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049485.187035:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049485.187039:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513382208:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049485.187040:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049485.187047:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382336:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049485.825345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049486.081268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049486.849356:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049487.105239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049487.873333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049488.129223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049488.897302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049489.153309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049489.921292:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049490.177214:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049490.305290:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049490.305297:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049490.305300:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049490.305309:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3126 00000100:00080000:3.0:1781049490.305313:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049490.305319:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781049490.305322:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513382400:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781049490.305328:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049490.305342:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513382528/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049490.305356:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049490.305359:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049490.305387:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382528:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049490.305394:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513382464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049490.306918:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513382400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049490.306941:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513382464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049490.306981:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049490.306987:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049490.306992:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382528:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049490.945296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049491.201194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049491.969283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049492.225240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049492.993325:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049493.249211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049494.017250:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049494.273220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049495.041360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049495.297199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049495.425372:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049495.425380:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049495.425385:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049495.425417:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513382592:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049495.425421:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3131 00000100:00080000:1.0:1781049495.425428:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049495.425436:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049495.425461:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049495.425483:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513382720/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049495.425511:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049495.425515:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049495.425536:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049495.425548:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513382656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049495.425610:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382720:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049495.426810:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513382656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049495.426837:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513382592:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049495.427119:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049495.427173:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049495.427178:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382720:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049496.065301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049496.321234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049497.089452:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049497.345229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049498.113276:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049498.369220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049499.137343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049499.393243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049500.161298:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049500.417263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049500.545322:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049500.545329:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049500.545333:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049500.545343:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3136 00000100:00080000:3.0:1781049500.545346:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049500.545351:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049500.545360:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049500.545373:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513382912/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781049500.545385:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513382784:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049500.545394:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049500.545397:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049500.545430:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513382848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049500.545431:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382912:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049500.546930:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513382784:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049500.546983:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513382848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049500.547026:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049500.547033:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049500.547040:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513382912:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049501.185344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049501.441237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049502.209303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049502.465190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049503.233331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049503.489241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049504.257357:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049504.513215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049505.281323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049505.537212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049505.665313:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049505.665320:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049505.665323:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049505.665332:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3141 00000100:00080000:1.0:1781049505.665336:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049505.665342:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049505.665351:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513382976:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781049505.665352:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049505.665384:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a5a8ed39 x1867562513383104/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049505.665404:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049505.665407:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049505.665423:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513383040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049505.665442:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383104:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049505.666834:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513382976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049505.666857:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513383040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049505.666891:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049505.666896:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049505.666900:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383104:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049506.305280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049506.561190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049507.329310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049507.585234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049508.353243:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049508.609224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049509.377303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049509.633246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049510.401285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049510.657236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049510.785329:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049510.785335:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049510.785338:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049510.785348:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3146 00000100:00080000:2.0:1781049510.785351:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781049510.785360:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049510.785361:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049510.785371:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049510.785385:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c91281b8 x1867562513383296/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049510.785403:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049510.785406:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049510.785425:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049510.785449:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383296:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049510.786807:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049510.786841:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049510.786861:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049510.786867:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049510.786874:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383296:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049511.425310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049511.681288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049512.449253:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049512.705306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049513.473285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049513.729252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049514.497333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049514.753286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049515.521284:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049515.777253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049515.905298:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049515.905307:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049515.905311:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049515.905321:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3151 00000100:00100000:3.0:1781049515.905322:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049515.905324:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049515.905332:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049515.905342:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049515.905359:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513383488/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049515.905379:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049515.905381:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049515.905399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049515.905422:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383488:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049515.906535:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049515.906586:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049515.906608:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049515.906612:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049515.906617:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383488:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049516.545264:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049516.801236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049517.569271:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049517.825266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049518.593323:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049518.849183:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049519.617272:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049519.873240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049520.641313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049520.897283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049521.025314:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049521.025320:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049521.025323:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049521.025334:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3156 00000100:00080000:3.0:1781049521.025338:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049521.025346:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383552:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049521.025346:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049521.025357:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049521.025376:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513383680/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049521.025393:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049521.025395:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049521.025430:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049521.025445:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383680:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049521.026823:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383552:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049521.026871:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049521.026897:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049521.026902:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049521.026919:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383680:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049521.665263:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049521.921285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049522.689297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049522.945253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049523.713328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049523.969224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049524.737321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049524.993222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049525.761296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049526.017224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049526.145308:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781049526.145330:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383744:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049526.145337:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049526.145343:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049526.145350:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3162 00000100:00080000:2.0:1781049526.145353:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049526.145359:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049526.145368:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049526.145389:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2849ba3 x1867562513383872/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049526.145420:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049526.145423:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049526.145442:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049526.145514:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383872:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049526.146420:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513383808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049526.146423:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383744:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049526.146446:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049526.146450:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049526.146454:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513383872:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049526.785333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049527.041228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049527.809365:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049528.065225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049528.833273:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049529.089229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049529.857301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049530.113238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049530.881346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049531.137228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049531.265364:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049531.265371:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049531.265374:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049531.265383:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3167 00000100:00100000:2.0:1781049531.265384:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383936:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049531.265386:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049531.265394:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049531.265404:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049531.265420:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9891000 x1867562513384064/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049531.265440:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049531.265443:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049531.265462:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513384000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049531.265489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384064:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049531.266766:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513383936:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049531.266811:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513384000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049531.266843:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049531.266848:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049531.266852:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384064:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049531.905283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049532.161263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049532.929309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049533.185224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049533.953301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049534.209323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049534.977258:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049535.233221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049536.001291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049536.257244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049536.385343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049536.385350:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049536.385354:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049536.385364:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3172 00000100:00080000:1.0:1781049536.385367:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049536.385373:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049536.385378:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513384128:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781049536.385383:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049536.385405:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b12abf0d x1867562513384256/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049536.385447:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049536.385449:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049536.385470:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513384192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049536.385506:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384256:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049536.386798:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513384128:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049536.386852:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513384192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049536.386877:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049536.386882:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049536.386886:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384256:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049537.025280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049537.281270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049538.049298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049538.305243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049539.073281:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049539.329234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049540.097371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049540.353256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049541.121328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049541.377219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049541.505326:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049541.505336:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049541.505340:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049541.505352:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3177 00000100:00080000:1.0:1781049541.505356:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781049541.505361:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513384320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049541.505366:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049541.505379:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049541.505397:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513384448/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049541.505415:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049541.505419:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049541.505445:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513384384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049541.505478:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384448:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049541.506832:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513384320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049541.506868:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513384384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049541.506894:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049541.506900:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049541.506906:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384448:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049542.145358:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049542.401231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049543.169357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049543.425256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049544.193358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049544.449276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049545.217335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049545.473223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049546.241388:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049546.497332:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049546.625301:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049546.625324:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049546.625328:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049546.625337:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3182 00000100:00100000:0.0:1781049546.625338:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513384512:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049546.625340:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049546.625348:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049546.625360:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049546.625376:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513384640/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049546.625392:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049546.625395:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049546.625410:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513384576:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049546.625444:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384640:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049546.626999:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513384576:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049546.627012:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513384512:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049546.627076:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049546.627082:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049546.627087:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384640:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049547.265292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049547.521245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049548.289356:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049548.545190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049549.313305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049549.569237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049550.337305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049550.593261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049551.361274:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049551.617224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049551.745332:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049551.745339:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049551.745343:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049551.745354:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3187 00000100:00080000:3.0:1781049551.745357:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049551.745363:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049551.745373:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049551.745386:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513384704:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049551.745389:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513384832/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049551.745409:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049551.745412:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049551.745438:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384832:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049551.745452:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513384768:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049551.746706:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513384704:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049551.746773:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513384768:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049551.746798:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049551.746811:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049551.746815:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513384832:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049552.385305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049552.641281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049553.409321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049553.665256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049554.433337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049554.689294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049555.457289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049555.713249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049556.481306:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049556.737268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049556.865262:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049556.865268:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049556.865272:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049556.865300:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3192 00000100:00080000:3.0:1781049556.865304:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049556.865307:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513384896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049556.865313:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049556.865325:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049556.865342:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513385024/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049556.865359:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049556.865363:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049556.865406:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513384960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049556.865432:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385024:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049556.866765:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513384960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049556.866780:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513384896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049556.866787:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049556.866793:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049556.866808:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385024:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049557.505348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049557.761250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049558.529289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049558.785297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049559.553268:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049559.809286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049560.577334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049560.833265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049561.601399:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049561.857285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049561.985317:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049561.985326:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049561.985330:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049561.985341:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513385088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049561.985342:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3197 00000100:00080000:3.0:1781049561.985349:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049561.985358:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049561.985372:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049561.985394:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513385216/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049561.985420:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049561.985442:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049561.985445:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049561.985454:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513385152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049561.985477:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385216:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049561.987209:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513385088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049561.987237:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513385152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049561.987357:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781049561.987370:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781049561.987376:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385216:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049562.625329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049562.881284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049563.649305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049563.905309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049564.673280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049564.929249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049565.697315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049565.953324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049566.721286:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049566.977263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049567.105367:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049567.105376:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049567.105381:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049567.105393:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3203 00000100:00100000:0.0:1781049567.105394:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513385280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049567.105397:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049567.105406:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049567.105420:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049567.105442:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513385408/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049567.105471:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049567.105475:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049567.105498:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513385344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049567.105521:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385408:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049567.120561:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513385280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049567.120570:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513385344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049567.120571:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049567.120579:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049567.120587:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385408:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049567.745293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049568.001220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049568.769264:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049569.025269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049569.793306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049570.049278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049570.817272:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049571.073221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049571.841285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049572.097272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049572.225284:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049572.225290:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049572.225293:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049572.225297:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513385472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049572.225302:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3208 00000100:00080000:1.0:1781049572.225306:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049572.225311:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049572.225322:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049572.225337:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f819707f x1867562513385600/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049572.225353:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049572.225356:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049572.225370:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513385536:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049572.225405:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385600:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049572.229545:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513385472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049572.229551:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513385536:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049572.229556:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049572.229567:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049572.229595:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385600:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049572.865376:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049573.121335:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049573.889261:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049574.145323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049574.913382:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049575.169251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049575.937304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049576.193256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049576.961288:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049577.217311:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049577.345301:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049577.345308:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049577.345312:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049577.345323:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3213 00000100:00080000:3.0:1781049577.345327:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049577.345336:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513385664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049577.345336:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049577.345345:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049577.345360:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513385792/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049577.345380:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049577.345383:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049577.345421:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513385728:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049577.345431:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385792:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049577.346723:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513385664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049577.346739:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049577.346744:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049577.346749:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385792:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049577.346803:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513385728:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781049577.985315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049578.241223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049579.009311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049579.265265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049580.033368:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049580.289196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049581.057306:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049581.313284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049582.081345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049582.337258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049582.465380:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049582.465401:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049582.465404:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049582.465415:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3218 00000100:00080000:1.0:1781049582.465418:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781049582.465422:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513385856:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049582.465464:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049582.465474:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049582.465492:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513385984/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049582.465512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049582.465514:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049582.465535:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513385920:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049582.465563:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385984:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049582.466777:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513385856:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049582.466791:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513385920:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049582.466886:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049582.466891:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049582.466897:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513385984:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049583.105405:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049583.361265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049584.129301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049584.385277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049585.153334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049585.409238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049586.177297:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049586.433238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049587.201352:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049587.457263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049587.585366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049587.585376:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049587.585380:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049587.585393:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386048:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049587.585393:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3223 00000100:00080000:3.0:1781049587.585399:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049587.585407:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049587.585419:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049587.585439:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513386176/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049587.585463:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049587.585468:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049587.585507:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049587.585511:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386176:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049587.585517:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513386112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049587.586889:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781049587.586894:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513386112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049587.586895:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049587.586902:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386176:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049587.586915:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386048:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049588.225311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049588.481238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049589.249345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049589.505316:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049590.273334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049590.529193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049591.297297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049591.553253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049592.321403:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049592.577224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049592.705351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049592.705360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049592.705368:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049592.705381:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3228 00000100:00080000:0.0:1781049592.705385:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781049592.705387:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513386240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049592.705395:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049592.705407:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049592.705443:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513386368/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049592.705462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049592.705466:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049592.705494:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049592.705505:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386368:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049592.705512:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513386304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049592.706794:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513386304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049592.706814:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513386240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049592.706850:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049592.706859:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049592.706866:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386368:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049593.345329:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049593.601233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049594.369430:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049594.625281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049595.393292:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049595.649269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049596.417337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049596.673213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049597.441270:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049597.697266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049597.825325:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049597.825334:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049597.825338:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049597.825350:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3233 00000100:00080000:3.0:1781049597.825354:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049597.825357:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049597.825361:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049597.825371:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049597.825387:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513386560/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049597.825408:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049597.825411:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049597.825437:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513386496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049597.825450:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386560:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049597.826942:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049597.826976:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781049597.826976:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513386496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049597.826981:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049597.826985:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386560:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049598.465304:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049598.721298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049599.489383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049599.745228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049600.513305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049600.769281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049601.537261:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049601.793223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049602.561276:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049602.817250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049602.945272:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049602.945329:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1781049602.945330:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049602.945333:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049602.945341:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3238 00000100:00080000:3.0:1781049602.945345:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049602.945352:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049602.945361:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049602.945379:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513386752/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049602.945395:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049602.945398:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049602.945420:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513386688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049602.945466:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386752:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049602.946958:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049602.946969:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049602.946974:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049602.946979:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386752:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049602.947004:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513386688:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049603.585289:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049603.841265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049604.609350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049604.865278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049605.633274:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049605.889231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049606.657358:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049606.913292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049607.681298:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049607.937248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049608.065338:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049608.065349:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049608.065352:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049608.065364:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3243 00000100:00080000:3.0:1781049608.065367:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049608.065374:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049608.065384:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049608.065392:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049608.065400:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513386944/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049608.065442:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049608.065446:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049608.065474:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513386880:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049608.065490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386944:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049608.066883:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513386816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049608.066907:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781049608.066908:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513386880:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049608.066912:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049608.066917:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513386944:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049608.705333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049608.961243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049609.729305:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049609.985233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049610.753285:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049611.009205:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049611.777341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049612.033201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049612.801406:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049613.057235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049613.185301:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049613.185312:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049613.185315:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049613.185326:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3249 00000100:00080000:0.0:1781049613.185330:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781049613.185330:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513387008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049613.185336:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049613.185347:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049613.185363:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513387136/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049613.185412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049613.185416:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049613.185447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387136:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049613.185459:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513387072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049613.186899:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513387008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049613.186900:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513387072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049613.186904:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049613.186912:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049613.186919:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387136:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049613.825344:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049614.081208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049614.849299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049615.105252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049615.873450:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049616.129300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049616.897297:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049617.153209:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049617.921356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049618.177286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049618.305300:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049618.305307:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049618.305310:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049618.305322:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3254 00000100:00080000:0.0:1781049618.305325:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049618.305332:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049618.305342:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049618.305344:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513387200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049618.305364:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b24224a5 x1867562513387328/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049618.305386:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049618.305390:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049618.305415:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513387264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049618.305437:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387328:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049618.306729:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513387200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049618.306735:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513387264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049618.306735:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049618.306742:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049618.306748:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387328:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049618.945379:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049619.201203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049619.969311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049620.225346:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049620.993345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049621.249290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049622.017388:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049622.273293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049623.041335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049623.297260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049623.425336:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049623.425343:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049623.425346:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049623.425357:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3259 00000100:00100000:2.0:1781049623.425360:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049623.425361:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049623.425371:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049623.425380:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049623.425395:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513387520/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049623.425430:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049623.425433:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049623.425458:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387520:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049623.425463:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513387456:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049623.426807:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513387456:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049623.426822:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781049623.426824:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049623.426829:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049623.426836:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387520:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049624.065344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049624.321263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049625.089297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049625.345212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049626.113365:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049626.369251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049627.137330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049627.393221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049628.161313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049628.417232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049628.545310:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049628.545316:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049628.545319:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049628.545330:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3264 00000100:00080000:3.0:1781049628.545333:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049628.545341:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049628.545351:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049628.545366:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513387712/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781049628.545378:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049628.545396:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049628.545399:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049628.545424:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513387648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049628.545440:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387712:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781049628.547037:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781049628.547040:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049628.547043:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049628.547054:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387712:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049628.547105:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513387648:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781049629.185254:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049629.441234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049630.209314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049630.465258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049631.233355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049631.489197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049632.257276:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049632.513309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049633.281371:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049633.537296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049633.665352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049633.665376:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049633.665381:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049633.665394:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3269 00000100:00080000:3.0:1781049633.665398:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049633.665405:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049633.665409:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049633.665426:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049633.665450:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513387904/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049633.665478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049633.665499:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049633.665530:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513387840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049633.665554:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387904:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049633.667015:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049633.667039:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513387840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:02020000:0.0:1781049633.667068:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781049633.667078:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049633.667083:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049633.667088:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513387904:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049634.305250:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049634.561259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049635.329345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049635.585217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049636.353356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049636.609236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049637.377311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049637.633260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049638.401255:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049638.657216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049638.785356:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049638.785362:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049638.785365:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049638.785376:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3274 00000100:00080000:2.0:1781049638.785378:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781049638.785383:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049638.785386:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049638.785410:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049638.785426:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513388096/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049638.785443:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049638.785446:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049638.785477:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513388032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049638.785498:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388096:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049638.786838:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513388032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049638.786842:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513387968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049638.786851:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049638.786859:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049638.786868:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388096:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049639.425309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049639.681238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049640.449355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049640.705275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049641.473399:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049641.729247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049642.497283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049642.753223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049643.521347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049643.777234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049643.905343:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049643.905362:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049643.905366:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049643.905367:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513388160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049643.905376:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3279 00000100:00080000:0.0:1781049643.905379:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049643.905385:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049643.905394:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049643.905409:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513388288/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049643.905427:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049643.905430:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049643.905449:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513388224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049643.905489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388288:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049643.906768:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513388160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049643.906770:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513388224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049643.906813:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049643.906822:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049643.906830:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388288:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049644.545325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049644.801276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049645.569297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049645.825307:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049646.593314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049646.849208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049647.617319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049647.873272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049648.641298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049648.897305:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049649.025354:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049649.025362:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049649.025366:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049649.025379:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3284 00000100:00080000:0.0:1781049649.025383:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781049649.025390:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513388352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049649.025413:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049649.025423:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049649.025440:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513388480/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049649.025459:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049649.025464:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049649.025491:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513388416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049649.025504:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388480:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049649.026703:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513388352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049649.026746:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513388416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049649.026762:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049649.026776:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049649.026782:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388480:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049649.665314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049649.921277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049650.689212:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781049650.945212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049651.713268:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049651.969231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049652.737329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049652.993233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049653.761278:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049654.017228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049654.145327:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049654.145335:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049654.145339:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049654.145351:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3290 00000100:00100000:3.0:1781049654.145352:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513388544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049654.145370:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049654.145378:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049654.145390:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049654.145408:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e5c8472 x1867562513388672/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049654.145432:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049654.145437:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049654.145464:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513388608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049654.145489:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388672:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049654.146622:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513388544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049654.146637:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513388608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049654.146825:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049654.146830:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049654.146835:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388672:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049654.785352:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049655.041232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049655.809335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049656.065213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049656.833326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049657.089231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049657.857318:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049658.113166:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049658.881282:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049659.137245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049659.265239:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049659.265244:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049659.265248:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049659.265257:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3295 00000100:00080000:2.0:1781049659.265261:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049659.265266:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049659.265276:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049659.265282:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513388736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049659.265290:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513388864/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049659.265306:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049659.265321:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049659.265341:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513388800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049659.265382:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388864:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049659.266656:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513388800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049659.266729:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513388736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049659.266828:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049659.266832:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049659.266836:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513388864:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049659.905293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049660.161235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049660.929334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049661.185240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049661.953347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049662.209267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049662.977288:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049663.233240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049664.001309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049664.257242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049664.385395:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049664.385404:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049664.385408:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049664.385422:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3300 00000100:00080000:0.0:1781049664.385426:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781049664.385431:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513388928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049664.385437:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049664.385450:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049664.385467:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513389056/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049664.385489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049664.385492:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049664.385516:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513388992:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049664.385534:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389056:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049664.386782:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513388928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049664.386806:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513388992:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049664.386807:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049664.386814:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049664.386820:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389056:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049665.025337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049665.281267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049666.049293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049666.305234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049667.073286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049667.329275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049668.097357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049668.353224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049669.121294:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049669.377271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049669.505330:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049669.505338:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049669.505341:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049669.505353:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3305 00000100:00080000:3.0:1781049669.505356:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049669.505366:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049669.505370:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049669.505382:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049669.505396:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000048c6b8a x1867562513389248/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049669.505413:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049669.505435:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049669.505453:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389248:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049669.505483:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049669.506777:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049669.506841:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049669.506870:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049669.506875:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049669.506879:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389248:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049670.145404:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049670.401195:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049671.169277:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049671.425251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049672.193353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049672.449248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049673.217345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049673.473206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049674.241332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049674.497297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049674.625314:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049674.625321:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049674.625324:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049674.625336:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3310 00000100:00080000:2.0:1781049674.625339:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781049674.625344:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389312:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049674.625367:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049674.625376:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049674.625393:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513389440/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049674.625411:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049674.625414:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049674.625447:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049674.625455:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513389376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049674.625461:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389440:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781049674.626678:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049674.626683:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049674.626683:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513389376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049674.626691:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389440:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049674.626696:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389312:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781049675.265288:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049675.521203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049676.289353:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049676.545286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049677.313347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049677.569234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049678.337395:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049678.593194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049679.361322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049679.617185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049679.745346:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049679.745353:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049679.745355:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049679.745367:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3315 00000100:00080000:2.0:1781049679.745370:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049679.745376:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049679.745386:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049679.745402:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c3ac388 x1867562513389632/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049679.745419:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049679.745422:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049679.745439:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049679.745452:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049679.745466:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389632:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049679.746801:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049679.746823:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389504:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049679.746833:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049679.746842:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049679.746851:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389632:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049680.385355:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049680.641258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049681.409256:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049681.665254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049682.433296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049682.689268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049683.457303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049683.713271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049684.481317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049684.737263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049684.865384:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049684.865390:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049684.865394:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049684.865404:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3320 00000100:00080000:3.0:1781049684.865408:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049684.865414:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049684.865423:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049684.865436:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049684.865438:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000288f7157 x1867562513389824/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049684.865468:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049684.865490:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049684.865547:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049684.865554:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389824:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049684.866772:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049684.866788:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049684.866822:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049684.866829:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049684.866836:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513389824:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049685.505336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049685.761281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049686.529296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049686.785234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049687.553359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049687.809301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049688.577303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049688.833217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049689.601366:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049689.857251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049689.985326:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049689.985332:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049689.985336:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049689.985347:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3325 00000100:00080000:3.0:1781049689.985351:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049689.985357:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049689.985367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049689.985376:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049689.985384:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513390016/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049689.985404:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049689.985407:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049689.985432:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049689.985437:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781049689.987041:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781049689.987041:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513389888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049689.987047:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049689.987054:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049689.987066:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513389952:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781049690.625295:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049690.881198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049691.649319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049691.905245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049692.673309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049692.929237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049693.697340:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049693.953263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049694.721308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049694.977225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049695.105448:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049695.105455:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049695.105458:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049695.105469:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3331 00000100:00080000:2.0:1781049695.105472:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049695.105478:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781049695.105486:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513390080:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:2.0:1781049695.105506:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049695.105525:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb943f69 x1867562513390208/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049695.105543:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049695.105546:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049695.105585:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513390144:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049695.105593:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390208:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049695.106937:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513390080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049695.106950:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513390144:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049695.106955:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049695.106960:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049695.106966:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390208:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049695.745395:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049696.001235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049696.769346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049697.025218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049697.793371:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049698.049276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049698.817373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049699.073276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049699.841375:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049700.097278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049700.225392:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049700.225401:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049700.225405:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049700.225417:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3336 00000100:00080000:2.0:1781049700.225422:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049700.225428:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781049700.225432:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513390272:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:2.0:1781049700.225438:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049700.225454:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084e7220c x1867562513390400/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049700.225477:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049700.225481:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049700.225513:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513390336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049700.225538:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390400:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049700.226897:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513390272:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049700.226898:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513390336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049700.226918:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049700.226924:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049700.226931:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390400:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049700.865338:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049701.121275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049701.889342:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049702.145296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049702.913301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049703.169312:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049703.937334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049704.193303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049704.961303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049705.217246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049705.345363:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049705.345372:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049705.345377:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049705.345391:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3341 00000100:00100000:1.0:1781049705.345391:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513390464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049705.345397:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049705.345406:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049705.345418:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049705.345436:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513390592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049705.345463:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049705.345468:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049705.345496:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513390528:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049705.345565:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390592:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049705.346736:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513390528:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049705.346736:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513390464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049705.346901:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049705.346906:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049705.346911:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390592:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049705.985319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049706.241272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049707.009305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049707.265258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049708.033302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049708.289260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049709.057390:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049709.313233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049710.081314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049710.337241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049710.465306:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049710.465314:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049710.465318:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049710.465331:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3346 00000100:00080000:0.0:1781049710.465335:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049710.465344:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781049710.465345:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513390656:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1781049710.465356:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049710.465375:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513390784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049710.465400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049710.465409:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049710.465434:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513390720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049710.465442:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049710.466787:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513390656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049710.466805:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513390720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049710.467003:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049710.467009:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049710.467013:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390784:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049711.105357:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049711.361270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049712.129363:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049712.385226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049713.153337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049713.409262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049714.177319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049714.433223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049715.201337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049715.457269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049715.585379:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049715.585389:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049715.585392:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049715.585403:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3351 00000100:00080000:2.0:1781049715.585406:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049715.585412:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781049715.585417:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513390848:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:2.0:1781049715.585421:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049715.585436:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513390976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049715.585457:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049715.585477:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049715.585502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390976:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049715.585507:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513390912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049715.586802:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513390912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049715.586807:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513390848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049715.589775:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049715.589788:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049715.589798:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513390976:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049716.225325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049716.481222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049717.249338:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049717.505279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049718.273379:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049718.529194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049719.297295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049719.553256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049720.321321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049720.577250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049720.705316:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049720.705322:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049720.705325:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049720.705336:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3356 00000100:00080000:2.0:1781049720.705339:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781049720.705346:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513391040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049720.705347:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049720.705357:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049720.705371:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e5c8472 x1867562513391168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049720.705386:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049720.705388:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049720.705409:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513391104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049720.705426:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391168:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049720.706681:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513391104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049720.706699:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513391040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049720.706705:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049720.706710:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049720.706720:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391168:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049721.345335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049721.601196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049722.369310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049722.625215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049723.393323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049723.649237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049724.417245:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049724.673218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049725.441276:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049725.697253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049725.825315:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049725.825321:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049725.825324:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049725.825335:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3361 00000100:00080000:0.0:1781049725.825338:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049725.825344:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049725.825353:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049725.825354:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513391232:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049725.825371:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513391360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049725.825397:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049725.825401:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049725.825426:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513391296:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049725.825442:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391360:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049725.826873:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513391232:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049725.826879:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049725.826881:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513391296:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049725.826885:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049725.826891:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391360:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049726.465304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049726.721233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049727.489317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049727.745262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049728.513313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049728.769229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049729.537257:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049729.793265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049730.561308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049730.817216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049730.945345:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049730.945351:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049730.945355:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049730.945366:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3366 00000100:00080000:3.0:1781049730.945369:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049730.945379:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513391488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049730.945394:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049730.945403:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049730.945418:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513391552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049730.945433:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049730.945435:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049730.945465:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391552:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049730.945472:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513391424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049730.946860:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513391488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049730.946860:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513391424:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049730.946876:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049730.946880:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049730.946884:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391552:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049731.585245:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049731.841225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049732.609285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049732.865288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049733.633320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049733.889206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049734.657316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049734.913277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049735.681323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049735.937189:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049736.065343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049736.065350:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049736.065353:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049736.065365:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3371 00000100:00080000:0.0:1781049736.065368:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049736.065375:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781049736.065377:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513391616:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1781049736.065384:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049736.065399:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513391744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049736.065417:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049736.065420:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049736.065441:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513391680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049736.065468:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391744:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049736.066864:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513391616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049736.066876:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049736.066882:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049736.066884:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513391680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049736.066889:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391744:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049736.705314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049736.961257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049737.729290:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049737.985288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049738.753350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049739.009233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049739.777270:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049740.033265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049740.801314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049741.057251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049741.185288:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049741.185296:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049741.185316:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049741.185324:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3377 00000100:00100000:1.0:1781049741.185325:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513391808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049741.185329:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049741.185335:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049741.185347:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049741.185363:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513391936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049741.185389:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049741.185394:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049741.185414:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513391872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049741.185447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391936:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049741.186767:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513391808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049741.186840:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513391872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049741.186840:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781049741.186846:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781049741.186853:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513391936:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049741.825311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049742.081231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049742.849295:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049743.105258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049743.873288:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049744.129238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049744.897285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049745.153249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049745.921269:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049746.177198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049746.305352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049746.305357:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049746.305361:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049746.305372:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3382 00000100:00080000:3.0:1781049746.305375:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049746.305380:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049746.305390:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049746.305393:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049746.305404:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513392128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049746.305422:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049746.305425:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049746.305455:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392064:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049746.305459:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392128:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049746.306705:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049746.306715:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392064:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049746.306738:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049746.306744:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049746.306750:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392128:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049746.945300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049747.201256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049747.969331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049748.225216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049748.993257:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049749.249248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049750.017337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049750.273220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049751.041255:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049751.297219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049751.425263:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781049751.425267:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049751.425270:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049751.425275:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049751.425281:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3387 00000100:00080000:3.0:1781049751.425284:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049751.425289:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049751.425301:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049751.425315:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513392320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049751.425332:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049751.425335:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049751.425353:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049751.425378:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392320:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049751.426508:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049751.426509:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049751.426746:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049751.426751:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049751.426757:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392320:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049752.065309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049752.321241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049753.089326:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049753.345207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049754.113328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049754.369248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049755.137301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049755.393247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049756.161339:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049756.417284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049756.545303:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049756.545336:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049756.545341:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781049756.545345:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392384:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049756.545351:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3392 00000100:00080000:2.0:1781049756.545356:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781049756.545361:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049756.545373:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049756.545391:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513392512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049756.545409:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049756.545412:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049756.545444:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049756.545463:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392512:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049756.546745:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049756.546747:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392384:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049756.546780:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049756.546787:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049756.546793:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392512:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049757.185336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049757.441224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049758.209390:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049758.465251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049759.233322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049759.489227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049760.257283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049760.513269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049761.281347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049761.537257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049761.665328:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049761.665334:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049761.665338:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049761.665348:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3397 00000100:00080000:3.0:1781049761.665351:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049761.665354:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049761.665360:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049761.665385:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049761.665401:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513392704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049761.665419:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049761.665422:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049761.665461:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049761.665468:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392704:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049761.666773:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049761.666774:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049761.666786:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049761.666791:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049761.666799:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392704:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049762.305258:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049762.561276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049763.329328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049763.585271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049764.353282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049764.609266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049765.377320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049765.633223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049766.401336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049766.657216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049766.785327:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049766.785334:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049766.785339:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049766.785350:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3402 00000100:00080000:3.0:1781049766.785354:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049766.785360:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049766.785372:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049766.785377:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049766.785390:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513392896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049766.785408:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049766.785411:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049766.785431:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049766.785490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392896:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049766.787322:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513392832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049766.787465:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049766.787568:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049766.787574:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049766.787579:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513392896:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049767.425384:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049767.681206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049768.449270:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049768.705230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049769.473318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049769.729212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049770.497258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049770.753229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049771.521304:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049771.777242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049771.905300:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049771.905320:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049771.905323:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049771.905334:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3407 00000100:00080000:3.0:1781049771.905337:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049771.905350:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392960:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049771.905356:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049771.905367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049771.905383:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513393088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049771.905398:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049771.905401:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049771.905435:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393088:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049771.905440:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513393024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049771.906605:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513393024:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049771.906656:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513392960:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049771.906753:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049771.906759:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049771.906779:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393088:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049772.545319:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049772.801197:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049773.569306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049773.825240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049774.593344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049774.849250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049775.617277:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049775.873242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049776.641302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049776.897259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049777.025344:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049777.025352:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049777.025357:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049777.025368:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3412 00000100:00080000:0.0:1781049777.025372:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049777.025378:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049777.025391:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049777.025410:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513393280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781049777.025413:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049777.025441:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049777.025446:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049777.025469:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513393216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049777.025487:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393280:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049777.026784:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049777.026788:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513393216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049777.026797:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049777.026803:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049777.026810:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393280:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049777.665343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049777.921247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049778.689321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049778.945251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049779.713303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049779.969245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049780.737282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049780.993212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049781.761266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049782.017292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049782.145371:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049782.145382:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049782.145386:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049782.145401:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3418 00000100:00080000:3.0:1781049782.145406:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049782.145411:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513393344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049782.145416:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049782.145429:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049782.145478:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513393472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049782.145503:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049782.145508:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049782.145522:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049782.145532:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513393408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049782.145537:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393472:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781049782.146892:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049782.146898:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049782.146905:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393472:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049782.146905:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513393408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049782.146925:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513393344:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781049782.785321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049783.041248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049783.809322:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049784.065259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049784.833332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049785.089206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049785.857292:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049786.113265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049786.881356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049787.137202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049787.265397:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049787.265407:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049787.265410:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049787.265424:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3423 00000100:00080000:0.0:1781049787.265428:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049787.265436:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049787.265448:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049787.265462:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049787.265470:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513393664/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049787.265496:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049787.265501:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049787.265533:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393664:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049787.265536:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513393600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049787.267038:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:3.0:1781049787.267058:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00100000:0.0:1781049787.267059:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513393600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049787.267070:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049787.267075:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049787.267082:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393664:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049787.905277:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049788.161256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049788.929331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049789.185274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049789.953314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049790.209216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049790.977304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049791.233255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049792.001282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049792.257212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049792.385346:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049792.385357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049792.385361:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049792.385376:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049792.385377:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3428 00000100:00080000:0.0:1781049792.385383:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049792.385393:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049792.385405:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049792.385445:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513393856/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049792.385471:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049792.385476:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049792.385497:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049792.385506:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513393792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049792.385525:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393856:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049792.386752:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513393792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049792.386769:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049792.386879:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049792.386888:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049792.386896:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513393856:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049793.025337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049793.281247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049794.049363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049794.305270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049795.073350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049795.329267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049796.097285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049796.353221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049797.121353:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049797.377251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049797.505272:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049797.505278:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049797.505281:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049797.505292:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3433 00000100:00080000:0.0:1781049797.505295:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049797.505301:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781049797.505309:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393920:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781049797.505311:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049797.505328:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513394048/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049797.505344:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049797.505347:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049797.505367:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513393984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049797.505400:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394048:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049797.506658:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513393920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049797.506700:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513393984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049797.516066:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049797.516075:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049797.516082:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394048:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049798.145316:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049798.401273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049799.169348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049799.425259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049800.193264:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049800.449201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049801.217351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049801.473285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049802.241341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049802.497257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049802.625339:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049802.625349:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049802.625352:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781049802.625370:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049802.625383:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3438 00000100:00080000:0.0:1781049802.625387:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049802.625395:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049802.625407:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049802.625430:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513394240/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049802.625449:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049802.625452:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049802.625476:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394240:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049802.625478:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049802.625487:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513394176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049802.626759:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513394176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049802.626762:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049802.626767:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049802.626772:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394240:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049802.626782:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394112:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781049803.265300:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049803.521281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049804.289352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049804.545234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049805.313294:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049805.569283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049806.337376:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049806.593274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049807.361338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049807.617267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049807.745300:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049807.745308:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049807.745311:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049807.745323:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3443 00000100:00080000:3.0:1781049807.745326:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781049807.745333:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049807.745343:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049807.745346:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513394304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049807.745359:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513394432/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049807.745377:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049807.745386:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049807.745419:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513394368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049807.745428:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394432:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049807.746859:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513394368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049807.746873:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513394304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049807.746877:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049807.746883:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049807.746888:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394432:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049808.385372:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049808.641238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049809.409339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049809.665231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049810.433381:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049810.689300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049811.457293:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049811.713246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049812.481360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049812.737298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049812.865376:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049812.865385:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049812.865389:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049812.865402:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3448 00000100:00100000:0.0:1781049812.865403:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049812.865406:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049812.865413:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049812.865423:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049812.865439:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001d23a43b x1867562513394624/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049812.865461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049812.865465:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049812.865497:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513394560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049812.865510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394624:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049812.867019:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049812.867031:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513394560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049812.867041:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049812.867047:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049812.867054:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394624:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049813.505326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049813.761245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049814.529348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049814.785258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049815.553318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049815.809307:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049816.577389:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049816.833319:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049817.601309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049817.857279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049817.985298:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049817.985304:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049817.985308:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049817.985316:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3453 00000100:00080000:1.0:1781049817.985320:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049817.985326:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049817.985336:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049817.985350:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084471047 x1867562513394816/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049817.985370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049817.985373:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049817.985389:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049817.985398:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513394752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049817.985436:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394816:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781049817.987076:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049817.987078:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049817.987086:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049817.987099:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513394816:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049817.987120:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513394752:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049818.625297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049818.881241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049819.649336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049819.905282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049820.673304:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049820.929241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049821.697318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049821.953256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049822.721328:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049822.977234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049823.105332:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049823.105339:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049823.105342:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049823.105354:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3459 00000100:00080000:0.0:1781049823.105358:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781049823.105365:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049823.105375:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049823.105382:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049823.105390:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513395008/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049823.105406:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049823.105409:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049823.105447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049823.105449:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513394944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049823.106785:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513394880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049823.106790:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049823.106793:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513394944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049823.106795:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049823.106803:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395008:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049823.745344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049824.001211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049824.769283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049825.025252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049825.793331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049826.049249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049826.817327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049827.073247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049827.841367:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049828.097276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049828.225288:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049828.225294:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049828.225297:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049828.225307:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3464 00000100:00080000:3.0:1781049828.225310:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781049828.225381:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513395072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049828.225396:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049828.225410:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049828.225427:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513395200/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049828.225444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049828.225447:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049828.225478:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395200:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049828.225507:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049828.225521:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513395136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049828.226490:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513395136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049828.226537:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513395072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049828.226560:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049828.226566:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049828.226573:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395200:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049828.865366:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049829.121242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049829.889374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049830.145270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049830.913369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049831.169236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049831.937357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049832.193297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049832.961331:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049833.217262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049833.345377:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049833.345385:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049833.345390:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049833.345404:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3469 00000100:00100000:0.0:1781049833.345409:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049833.345414:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049833.345427:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049833.345441:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049833.345462:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c26c4b8c x1867562513395392/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049833.345488:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049833.345491:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049833.345524:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049833.345528:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049833.346868:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049833.346874:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049833.346880:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049833.346885:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049833.346940:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395328:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049833.985348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049834.241229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049835.009297:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049835.265254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049836.033380:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049836.289289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049837.057330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049837.313235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049838.081303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049838.337232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049838.465258:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781049838.465273:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049838.465279:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049838.465284:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049838.465290:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3474 00000100:00080000:1.0:1781049838.465293:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049838.465299:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049838.465309:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049838.465326:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b98b0b8e x1867562513395584/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049838.465343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049838.465345:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049838.465363:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395456:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049838.465451:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395584:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049838.466526:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395456:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049838.466527:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049838.466655:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049838.466660:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049838.466665:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395584:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049839.105295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049839.361230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049840.129322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049840.385251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049841.153313:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049841.409249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049842.177314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049842.433275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049843.201372:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049843.457212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049843.585290:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049843.585296:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049843.585299:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049843.585309:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3479 00000100:00080000:1.0:1781049843.585312:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049843.585319:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049843.585329:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049843.585350:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049843.585364:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b738af86 x1867562513395776/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049843.585383:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049843.585387:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049843.585433:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049843.585434:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049843.586829:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049843.586859:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049843.586866:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049843.586871:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049843.586876:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395776:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049844.225289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049844.481253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049845.249359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049845.505382:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049846.273316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049846.529263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049847.297346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049847.553266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049848.321309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049848.577228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049848.705266:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049848.705272:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049848.705276:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049848.705286:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3484 00000100:00100000:0.0:1781049848.705290:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049848.705290:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781049848.705298:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049848.705309:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049848.705324:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513395968/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049848.705340:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049848.705343:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049848.705376:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395968:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049848.705391:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049848.706662:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513395840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049848.706673:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513395904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049848.706691:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049848.706696:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049848.706702:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513395968:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049849.345282:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049849.601227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049850.369290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049850.625201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049851.393279:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049851.649190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049852.417275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049852.673199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049853.441315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049853.697238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049853.825263:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049853.825269:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049853.825273:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049853.825283:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3489 00000100:00080000:3.0:1781049853.825287:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 6s 00000100:00080000:3.0:1781049853.825293:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781049853.825303:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396032:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781049853.825303:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049853.825319:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9891000 x1867562513396160/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049853.825353:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049853.825356:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049853.825390:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513396096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049853.825404:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396160:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049853.826534:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049853.826554:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049853.826559:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049853.826565:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396160:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049853.826578:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513396096:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049854.465240:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049854.721250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049855.489326:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049855.745234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049856.513283:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049856.769253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049857.537300:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049857.793236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049858.561321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049858.817215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049858.945339:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049858.945348:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049858.945353:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049858.945365:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3494 00000100:00100000:2.0:1781049858.945366:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049858.945371:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 11s 00000100:00080000:3.0:1781049858.945380:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049858.945393:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049858.945413:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f9891000 x1867562513396352/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049858.945450:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049858.945455:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049858.945467:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049858.945473:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513396288:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049858.945532:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396352:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049858.946765:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513396288:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049858.946782:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049858.946798:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049858.946802:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049858.946807:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396352:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049859.585354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049859.841243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049860.609346:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049860.865241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049861.633254:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049861.889222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049862.657296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049862.913195:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049863.681278:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049863.937198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049864.065354:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049864.065363:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049864.065368:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049864.065381:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3499 00000100:00100000:0.0:1781049864.065382:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513396416:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049864.065398:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1781049864.065419:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049864.065431:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049864.065448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513396544/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049864.065469:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049864.065473:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049864.065490:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049864.065502:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513396480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049864.065503:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396544:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781049864.066861:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049864.066866:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049864.066870:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513396480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049864.066880:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513396416:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049864.066886:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396544:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049864.705316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049864.961210:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049865.729309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049865.985218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049866.753342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049867.009199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049867.777364:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049868.033195:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049868.801286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049869.057249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049869.185299:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049869.185309:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1781049869.185337:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049869.185433:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781049869.185438:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513396672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049869.185492:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049869.185502:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3505 00000100:00080000:3.0:1781049869.185506:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781049869.185513:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049869.185525:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049869.185545:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513396736/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049869.185569:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049869.185573:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049869.185600:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396736:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781049869.186573:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049869.186577:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049869.186582:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396736:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049869.186586:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513396672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049869.186605:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396608:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781049869.825270:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049870.081195:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049870.849352:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049871.105232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049871.873343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049872.129230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049872.897371:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049873.153260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049873.921347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049874.177238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049874.305390:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049874.305397:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049874.305400:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049874.305411:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3510 00000100:00080000:3.0:1781049874.305414:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781049874.305420:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049874.305431:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049874.305433:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049874.305446:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bb64ba62 x1867562513396928/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049874.305479:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049874.305482:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049874.305518:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396928:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049874.305522:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513396864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049874.306751:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049874.306753:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049874.306764:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049874.306768:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513396864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049874.306771:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513396928:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049874.945326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049875.201286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049875.969277:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049876.225291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049876.993362:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049877.249301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049878.017403:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049878.273253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049879.041308:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049879.297274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049879.425330:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049879.425336:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049879.425340:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049879.425352:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3515 00000100:00080000:2.0:1781049879.425356:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781049879.425360:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049879.425364:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049879.425376:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049879.425412:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2849ba3 x1867562513397120/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049879.425428:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049879.425431:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049879.425454:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513397056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049879.425492:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397120:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049879.426614:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513397056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049879.426627:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513396992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049879.426639:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049879.426646:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049879.426653:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397120:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049880.065347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049880.321207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049881.089289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049881.345240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049882.113310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049882.369218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049883.137283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049883.393245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049884.161350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049884.417259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049884.545298:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049884.545305:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049884.545308:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049884.545319:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3520 00000100:00080000:2.0:1781049884.545322:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049884.545329:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049884.545338:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049884.545352:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513397312/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781049884.545368:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513397184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049884.545373:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049884.545378:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049884.545402:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513397248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049884.545419:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397312:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049884.546728:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513397184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049884.546729:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513397248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049884.546746:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049884.546752:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049884.546757:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397312:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049885.185321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049885.441240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049886.209349:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049886.465212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049887.233347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049887.489219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049888.257349:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049888.513249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049889.281322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049889.537213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049889.665391:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049889.665400:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049889.665403:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049889.665416:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3525 00000100:00080000:1.0:1781049889.665420:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049889.665430:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049889.665437:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513397376:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781049889.665458:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049889.665481:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d7e018b1 x1867562513397504/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049889.665509:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049889.665512:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049889.665545:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513397440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049889.665573:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397504:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049889.667607:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513397440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049889.667614:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513397376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049889.667651:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049889.667660:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049889.667670:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397504:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781049890.305390:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049890.561257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049891.329374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049891.585268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049892.353350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049892.609294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049893.377280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049893.633276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049894.401283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049894.657216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049894.785370:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049894.785380:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049894.785383:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049894.785396:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3530 00000100:00080000:1.0:1781049894.785400:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049894.785406:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049894.785407:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513397568:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781049894.785416:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049894.785432:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a161b523 x1867562513397696/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049894.785453:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049894.785457:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049894.785493:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397696:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049894.785500:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513397632:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049894.787194:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781049894.787197:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513397568:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049894.787198:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513397632:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049894.787200:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049894.787209:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397696:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049895.425276:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049895.681270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049896.449309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049896.705288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049897.473281:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049897.729220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049898.497296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049898.753272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049899.521347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049899.777248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049899.905378:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049899.905386:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049899.905389:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049899.905400:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3535 00000100:00080000:2.0:1781049899.905404:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049899.905411:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049899.905420:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049899.905435:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c170795f x1867562513397888/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781049899.905444:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513397760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049899.905457:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049899.905462:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049899.905498:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513397824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049899.905511:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397888:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049899.907352:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513397824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049899.907354:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513397760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049899.907401:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049899.907408:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049899.907414:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513397888:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049900.545292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049900.801283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049901.569437:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049901.825236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049902.593324:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049902.849247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049903.617359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049903.873259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049904.641345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049904.897225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049905.025378:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049905.025389:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049905.025394:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049905.025409:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3540 00000100:00080000:2.0:1781049905.025416:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049905.025429:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049905.025443:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049905.025467:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f57815d x1867562513398080/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781049905.025474:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513397952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049905.025517:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049905.025522:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049905.025558:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513398016:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049905.025585:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398080:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049905.027733:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513397952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049905.027804:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513398016:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049905.027835:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049905.027841:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049905.027847:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398080:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049905.665442:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049905.921246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049906.689352:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049906.945237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049907.713387:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049907.969199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049908.737384:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049908.993316:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049909.761389:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049910.017277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049910.145315:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049910.145321:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049910.145337:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049910.145347:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3546 00000100:00080000:1.0:1781049910.145350:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049910.145356:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781049910.145361:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513398144:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781049910.145366:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049910.145382:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ea0bf1d5 x1867562513398272/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049910.145410:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049910.145414:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049910.145437:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513398208:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049910.145518:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398272:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781049910.146808:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513398144:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049910.146833:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513398208:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049910.146857:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049910.146863:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049910.146868:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398272:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049910.785415:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049911.041331:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049911.809350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049912.065272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049912.833367:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049913.089211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049913.857307:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049914.113290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049914.881273:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049915.137255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049915.265400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049915.265408:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049915.265412:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049915.265424:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3551 00000100:00080000:2.0:1781049915.265428:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049915.265435:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049915.265446:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049915.265448:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513398336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781049915.265464:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f57815d x1867562513398464/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049915.265490:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049915.265494:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049915.265549:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513398400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049915.265589:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398464:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049915.266946:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513398400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049915.267072:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513398336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049915.267112:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049915.267118:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049915.267145:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398464:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049915.905275:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049916.161257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049916.929329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049917.185231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049917.953286:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049918.209218:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049918.977348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049919.233283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049920.001290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049920.257222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049920.385287:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049920.385294:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049920.385317:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781049920.385323:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513398528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049920.385327:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3556 00000100:00080000:2.0:1781049920.385331:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049920.385338:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049920.385347:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049920.385363:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c91281b8 x1867562513398656/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049920.385380:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049920.385383:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049920.385404:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513398592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049920.385434:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398656:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781049920.386680:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049920.386684:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049920.386687:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513398528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049920.386688:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513398592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049920.386698:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398656:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049921.025474:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049921.281217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049922.049344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049922.305283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049923.073422:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049923.329263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049924.097371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049924.353274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049925.121316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049925.377300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049925.505364:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049925.505374:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049925.505380:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049925.505395:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3561 00000100:00080000:1.0:1781049925.505400:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049925.505408:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049925.505420:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049925.505427:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513398720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049925.505447:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000013bcd30c x1867562513398848/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049925.505514:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049925.505519:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049925.505558:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513398784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049925.505579:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398848:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049925.507246:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513398720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049925.507266:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513398784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049925.507289:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049925.507296:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049925.507303:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513398848:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781049926.145397:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049926.401329:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049927.169346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049927.425322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049928.193369:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049928.449216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049929.217423:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049929.473242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049930.241310:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049930.497212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049930.625363:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049930.625370:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049930.625373:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049930.625385:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3566 00000100:00080000:1.0:1781049930.625388:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049930.625395:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049930.625405:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049930.625418:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513398912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049930.625422:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e41ff85d x1867562513399040/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049930.625440:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049930.625443:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049930.625463:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513398976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049930.625499:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399040:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049930.626950:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513398912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049930.626950:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513398976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049930.627105:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049930.627108:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049930.627112:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399040:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049931.265337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049931.521275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049932.289285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049932.545241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049933.313345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049933.569274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049934.337350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049934.593298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049935.361402:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049935.617274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049935.745339:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049935.745349:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049935.745376:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049935.745386:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049935.745387:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3571 00000100:00080000:1.0:1781049935.745394:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049935.745403:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049935.745413:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049935.745449:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513399232/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049935.745469:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049935.745472:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049935.745492:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513399168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049935.745511:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399232:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049935.747599:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049935.747634:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513399168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049935.747653:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049935.747658:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049935.747663:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399232:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049936.385424:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049936.641227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049937.409397:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049937.665256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049938.433427:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049938.689235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049939.457266:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049939.713242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049940.481341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049940.737216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049940.865366:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049940.865376:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049940.865380:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781049940.865394:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781049940.865394:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3576 00000100:00080000:1.0:1781049940.865400:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049940.865409:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049940.865420:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049940.865440:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000000dbb310 x1867562513399424/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049940.865462:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049940.865466:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049940.865499:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781049940.865507:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513399360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049940.865526:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399424:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049940.866932:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513399360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049940.866945:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:3.0:1781049940.866975:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:3.0:1781049940.873850:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049940.873879:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049940.873890:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399424:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049941.505268:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049941.761237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049942.529339:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049942.785303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049943.553317:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049943.809204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049944.577404:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049944.833255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049945.601353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049945.857216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049945.985421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049945.985431:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781049945.985435:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049945.985437:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049945.985448:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3581 00000100:00080000:0.0:1781049945.985455:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049945.985467:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049945.985480:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049945.985504:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513399616/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049945.985531:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049945.985536:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049945.985568:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513399552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049945.985593:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399616:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049945.987313:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049945.987314:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513399552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049945.987400:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049945.987405:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049945.987410:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399616:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049946.625420:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049946.881271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049947.649269:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049947.905281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049948.673352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049948.929294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049949.697351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049949.953262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049950.721333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049950.977254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049951.105393:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049951.105401:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049951.105405:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049951.105418:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3587 00000100:00080000:0.0:1781049951.105422:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781049951.105424:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049951.105429:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049951.105461:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049951.105495:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513399808/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049951.105520:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049951.105524:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049951.105544:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513399744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049951.105601:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399808:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049951.107088:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513399680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049951.107089:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513399744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049951.107156:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049951.107162:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049951.107167:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513399808:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781049951.745327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049952.001254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049952.769383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049953.025274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049953.793324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049954.049266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781049954.817330:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049955.073282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049955.841244:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049956.097226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049956.225380:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049956.225388:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049956.225391:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049956.225403:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3592 00000100:00080000:2.0:1781049956.225407:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781049956.225414:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513399872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781049956.225415:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049956.225426:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049956.225441:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513400000/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049956.225483:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049956.225487:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049956.225503:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513399936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049956.225528:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400000:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781049956.227381:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513399936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049956.227384:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513399872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049956.227417:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049956.227424:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049956.227431:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400000:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049956.865302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049957.121246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049957.889320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049958.145227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049958.913293:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049959.169241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049959.937321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049960.193198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049960.961302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049961.217244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049961.345345:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049961.345354:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049961.345358:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049961.345371:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3597 00000100:00080000:0.0:1781049961.345375:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781049961.345388:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513400064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049961.345389:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049961.345414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049961.345440:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513400192/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049961.345486:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049961.345490:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781049961.345509:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781049961.345521:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513400128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049961.345546:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400192:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781049961.346972:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513400128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049961.346982:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513400064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049961.347026:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049961.347033:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049961.347039:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400192:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049961.985305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049962.241228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049963.009286:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049963.265251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049964.033369:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049964.289283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049965.057306:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049965.313208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049966.081337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049966.337277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049966.465324:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049966.465332:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049966.465335:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049966.465347:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3602 00000100:00080000:2.0:1781049966.465349:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781049966.465357:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781049966.465360:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513400256:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781049966.465367:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049966.465380:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513400384/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049966.465397:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049966.465417:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781049966.465438:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513400320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781049966.465448:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400384:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781049966.466802:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781049966.466803:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513400256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049966.466804:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513400320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049966.466810:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049966.466819:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400384:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049967.105371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049967.361224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049968.129358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049968.385261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049969.153323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049969.409228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049970.177281:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049970.433249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049971.201302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049971.457244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049971.585378:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049971.585388:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049971.585392:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049971.585405:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3607 00000100:00080000:0.0:1781049971.585409:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1781049971.585414:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513400448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049971.585419:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049971.585433:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049971.585452:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513400576/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049971.585474:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049971.585476:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049971.585498:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513400512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049971.585549:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400576:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049971.586732:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513400512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049971.586745:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513400448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049971.586910:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049971.586915:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049971.586921:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400576:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049972.225315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049972.481228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049973.249311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049973.505255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049974.273347:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049974.529268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049975.297359:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049975.553222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049976.321352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049976.577283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781049976.705330:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049976.705350:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781049976.705354:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781049976.705364:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3612 00000100:00100000:1.0:1781049976.705365:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513400640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049976.705368:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781049976.705376:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781049976.705387:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781049976.705402:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003cae67cc x1867562513400768/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781049976.705434:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781049976.705437:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049976.705453:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513400704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781049976.705505:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400768:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781049976.706973:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513400704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781049976.706977:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513400640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049976.707019:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049976.707024:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049976.707028:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400768:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049977.345333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049977.601217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049978.369321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049978.625220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049979.393387:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049979.649306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049980.417389:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049980.673233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049981.441388:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049981.697192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781049981.825316:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781049981.825351:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513400832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781049981.825384:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049981.825389:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049981.825393:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781049981.825400:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3617 00000100:00080000:3.0:1781049981.825403:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781049981.825408:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781049981.825413:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049981.825420:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513400896:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:3.0:1781049981.825467:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049981.825486:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513400960/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049981.825512:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049981.825532:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781049981.825549:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400960:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781049981.826646:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513400832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781049981.826687:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513400896:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781049981.826712:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049981.826718:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049981.826725:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513400960:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781049982.465262:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049982.721206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049983.489390:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049983.745279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049984.513347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049984.769238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049985.537327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049985.793319:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049986.561342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049986.817298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781049986.945362:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781049986.945368:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781049986.945372:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781049986.945394:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3622 00000100:00080000:1.0:1781049986.945397:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781049986.945403:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781049986.945414:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781049986.945429:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513401152/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781049986.945448:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781049986.945454:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513401024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781049986.945455:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049986.945488:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401152:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781049986.945521:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513401088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049986.947168:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513401024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781049986.947196:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513401088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049986.947260:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781049986.947266:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781049986.947271:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401152:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049987.585345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049987.841244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049988.609299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049988.865249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049989.633386:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049989.889326:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049990.657340:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049990.913233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049991.681306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049991.937251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781049992.065351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049992.065359:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781049992.065363:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781049992.065375:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3627 00000100:00080000:2.0:1781049992.065379:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781049992.065381:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781049992.065388:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781049992.065400:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781049992.065418:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513401344/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781049992.065441:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781049992.065445:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049992.065470:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781049992.065476:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513401280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049992.065555:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401344:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781049992.066766:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513401280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049992.066776:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049992.067177:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049992.067197:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049992.067204:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401344:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781049992.705302:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049992.961249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049993.729318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049993.985250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049994.753266:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049995.009202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049995.777262:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049996.033229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049996.801335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049997.057187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781049997.185302:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049997.185308:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781049997.185325:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781049997.185328:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781049997.185334:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3633 00000100:00080000:3.0:1781049997.185337:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781049997.185342:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781049997.185352:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781049997.185369:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513401536/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781049997.185385:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781049997.185388:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781049997.185406:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513401472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781049997.185436:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401536:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781049997.186794:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781049997.186805:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513401472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781049997.186808:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781049997.186814:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781049997.186819:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401536:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781049997.825289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049998.081199:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781049998.849309:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781049999.105229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781049999.873251:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050000.129228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050000.897320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050001.153281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050001.921330:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050002.177292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050002.305340:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050002.305348:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050002.305352:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050002.305364:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3638 00000100:00080000:0.0:1781050002.305368:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050002.305375:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781050002.305382:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513401600:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781050002.305389:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050002.305409:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513401728/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050002.305431:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050002.305435:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050002.305473:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401728:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050002.305479:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513401664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050002.307213:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513401600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050002.307313:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513401664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050002.307348:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050002.307353:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050002.307357:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401728:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050002.945420:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050003.201283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050003.969309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050004.225232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050004.993361:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050005.249268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050006.017275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050006.273242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050007.041336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050007.297276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050007.425352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781050007.425381:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401792:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050007.425382:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050007.425386:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050007.425393:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3643 00000100:00080000:3.0:1781050007.425396:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050007.425402:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050007.425412:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050007.425429:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513401920/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050007.425449:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050007.425453:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050007.425478:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513401856:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050007.425481:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401920:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050007.426611:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401792:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050007.426656:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513401856:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050007.426717:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050007.426726:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050007.426730:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513401920:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050008.065332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050008.321281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050009.089382:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050009.345272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050010.113321:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050010.369229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050011.137305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050011.393249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050012.161283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050012.417239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050012.545315:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050012.545322:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050012.545325:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050012.545337:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3648 00000100:00080000:2.0:1781050012.545340:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781050012.545343:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402048:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050012.545349:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050012.545359:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050012.545376:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513402112/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050012.545392:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050012.545395:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050012.545430:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050012.545469:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402112:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050012.546804:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402048:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050012.546811:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513401984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050012.546824:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050012.546829:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050012.546834:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402112:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050013.185303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050013.441228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050014.209397:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050014.465196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050015.233318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050015.489300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050016.257362:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050016.513190:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050017.281298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050017.537226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050017.665337:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050017.665343:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050017.665346:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050017.665358:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3653 00000100:00080000:0.0:1781050017.665361:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050017.665367:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050017.665376:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050017.665391:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513402304/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781050017.665397:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513402176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050017.665410:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050017.665414:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050017.665436:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513402240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050017.665454:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402304:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050017.666878:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513402176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050017.666905:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513402240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050017.666911:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050017.666916:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050017.666922:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402304:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050018.305335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050018.561254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050019.329297:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050019.585208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050020.353310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050020.609255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050021.377264:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050021.633264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050022.401344:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050022.657286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050022.785335:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050022.785342:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050022.785346:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050022.785357:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3658 00000100:00080000:2.0:1781050022.785361:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050022.785367:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781050022.785376:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402368:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:2.0:1781050022.785377:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050022.785392:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513402496/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050022.785407:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050022.785410:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050022.785426:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050022.785437:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402496:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050022.786766:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050022.786771:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050022.786792:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050022.786798:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050022.786799:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402496:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050023.425296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050023.681198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050024.449296:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050024.705200:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050025.473284:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050025.729247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050026.497319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050026.753275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050027.521267:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050027.777321:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050027.905408:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050027.905416:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050027.905420:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050027.905444:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3663 00000100:00080000:2.0:1781050027.905448:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050027.905455:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050027.905481:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050027.905561:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c3ac388 x1867562513402688/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781050027.905575:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050027.905582:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050027.905587:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050027.905624:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050027.905646:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402688:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050027.907169:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050027.907171:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050027.907201:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050027.907207:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050027.907212:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402688:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050028.545371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050028.801288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050029.569363:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050029.825250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050030.593294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050030.849365:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050031.617317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050031.873239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050032.641375:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050032.897286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050033.025303:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050033.025321:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050033.025325:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781050033.025339:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050033.025350:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3668 00000100:00080000:3.0:1781050033.025353:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050033.025361:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050033.025369:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050033.025386:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513402880/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050033.025403:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050033.025405:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050033.025475:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402880:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050033.025480:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050033.026795:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513402816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050033.026799:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050033.026804:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050033.026809:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050033.026814:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513402880:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050033.665463:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050033.921222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050034.689464:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050034.945358:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050035.713270:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050035.969184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050036.737298:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050036.993279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050037.761357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050038.017443:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050038.145498:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050038.145508:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050038.145513:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050038.145541:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3674 00000100:00080000:3.0:1781050038.145546:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050038.145560:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050038.145573:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050038.145596:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513403072/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050038.145604:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402944:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050038.145653:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050038.145658:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050038.145741:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513403008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050038.145755:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403072:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050038.147701:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513402944:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050038.147702:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513403008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050038.147762:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050038.147771:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050038.147781:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403072:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050038.785425:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050039.041264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050039.809402:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050040.065265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050040.833302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050041.089241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050041.857356:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050042.113304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050042.881343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050043.137323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050043.265379:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050043.265389:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050043.265393:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050043.265407:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3679 00000100:00080000:3.0:1781050043.265412:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781050043.265413:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513403136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050043.265422:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050043.265434:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050043.265451:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513403264/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050043.265468:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050043.265471:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050043.265503:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513403200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050043.265510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403264:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050043.266809:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513403136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050043.266815:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513403200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050043.266827:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050043.266832:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050043.266840:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403264:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050043.905363:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050044.161324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050044.929340:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050045.185281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050045.953394:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050046.209321:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050046.977331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050047.233280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050048.001320:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050048.257275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050048.385290:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050048.385297:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050048.385301:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050048.385314:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3684 00000100:00080000:3.0:1781050048.385317:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050048.385323:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050048.385332:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050048.385345:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513403456/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050048.385358:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513403328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050048.385361:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050048.385370:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050048.385398:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403456:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050048.385399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513403392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050048.386719:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513403328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050048.386748:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781050048.386761:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513403392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050048.386769:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050048.386776:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403456:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050049.025334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050049.281301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050050.049388:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050050.305228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050051.073272:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050051.329233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050052.097299:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050052.353265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050053.121338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050053.377207:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050053.505350:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781050053.505374:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513403520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050053.505378:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050053.505383:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050053.505391:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3689 00000100:00080000:3.0:1781050053.505395:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050053.505404:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050053.505416:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050053.505439:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513403648/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050053.505462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050053.505467:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050053.505477:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050053.505485:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513403584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050053.505506:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403648:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050053.506935:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781050053.506939:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513403584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050053.506942:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050053.506951:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403648:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050053.506952:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513403520:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050054.145288:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050054.401254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050054.529320:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513403712:192.168.201.144@tcp:17:kworker/2:1.0 00000100:00100000:3.0:1781050054.530939:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513403712:192.168.201.144@tcp:17:kworker/2:1.0 00010000:00010000:3.0:1781050055.169332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050055.425297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050056.193261:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050056.449193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050057.217340:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050057.473250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050058.241335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050058.497212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050058.625373:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050058.625382:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050058.625387:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050058.625399:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3694 00000100:00100000:2.0:1781050058.625403:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513403776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050058.625425:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050058.625434:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050058.625444:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050058.625461:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000048c6b8a x1867562513403904/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050058.625478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050058.625483:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050058.625505:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513403840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050058.625509:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403904:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050058.626842:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513403840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050058.626843:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513403776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050058.626848:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050058.626853:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050058.626858:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513403904:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050059.265293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050059.521272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050060.289312:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050060.545191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050061.313341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050061.569272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050062.337316:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050062.593184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050063.361349:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050063.617291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050063.745290:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050063.745297:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050063.745300:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050063.745310:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3699 00000100:00080000:3.0:1781050063.745314:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050063.745320:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050063.745346:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050063.745363:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513404096/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050063.745370:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513403968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050063.745385:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050063.745411:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050063.745441:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404096:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050063.745460:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513404032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050063.746782:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513404032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050063.746792:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1781050063.746795:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513403968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050063.746798:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050063.746805:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404096:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050064.385332:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050064.641235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050065.409347:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050065.665280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050066.433296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050066.689271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050067.457311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050067.713244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050068.481305:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050068.737236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050068.865378:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050068.865388:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050068.865392:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781050068.865400:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513404160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050068.865404:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3704 00000100:00080000:3.0:1781050068.865407:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050068.865415:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050068.865435:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050068.865456:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000288f7157 x1867562513404288/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050068.865480:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050068.865484:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050068.865512:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513404224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050068.865544:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404288:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050068.866881:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513404160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050068.866990:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513404224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050068.867076:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050068.867080:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050068.867084:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404288:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050069.505310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050069.761203:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050070.529348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050070.785208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050071.553274:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050071.809243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050072.577381:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050072.833221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050073.601316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050073.857238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050073.985258:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050073.985265:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050073.985267:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050073.985279:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3709 00000100:00080000:1.0:1781050073.985282:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050073.985289:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050073.985299:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050073.985311:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050073.985315:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001cbc13b4 x1867562513404480/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050073.985332:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050073.985334:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050073.985360:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404480:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050073.985375:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050073.986725:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781050073.986727:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050073.986730:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050073.986739:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404480:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050073.986756:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404416:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050074.625381:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050074.881216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050075.649267:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050075.905250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050076.673380:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050076.929297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050077.697388:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050077.953234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050078.721286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050078.977289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050079.105276:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050079.105290:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050079.105294:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050079.105304:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3715 00000100:00080000:1.0:1781050079.105307:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050079.105313:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050079.105343:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050079.105360:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050079.105363:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513404672/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050079.105385:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050079.105389:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050079.105410:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050079.105441:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404672:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050079.107029:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404544:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050079.107044:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050079.107069:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050079.107075:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050079.107079:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404672:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050079.745322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050080.001206:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050080.769400:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050081.025278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050081.793328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050082.049278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050082.817379:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050083.073260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050083.841350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050084.097334:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050084.225349:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050084.225357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050084.225360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050084.225371:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3720 00000100:00080000:0.0:1781050084.225374:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050084.225382:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781050084.225388:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404800:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781050084.225390:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050084.225404:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513404864/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050084.225421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050084.225424:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050084.225471:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404736:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050084.225477:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404864:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050084.226715:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404800:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050084.226724:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404736:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050084.226899:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050084.226903:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050084.226907:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513404864:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050084.865374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050085.121264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050085.889276:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050086.145242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050086.913372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050087.169288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050087.937310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050088.193261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050088.961267:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050089.217260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050089.345404:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050089.345414:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050089.345417:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781050089.345419:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050089.345440:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3725 00000100:00080000:0.0:1781050089.345463:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050089.345469:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050089.345479:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050089.345495:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513405056/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050089.345512:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050089.345515:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050089.345537:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050089.345544:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405056:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050089.346904:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513404992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050089.346909:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050089.346914:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050089.346915:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513404928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050089.346921:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405056:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050089.985331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050090.241244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050091.009275:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050091.265240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050092.033344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050092.289224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050093.057271:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050093.313270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050094.081377:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050094.337278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050094.465298:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050094.465305:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050094.465308:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781050094.465316:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050094.465317:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3730 00000100:00080000:0.0:1781050094.465322:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050094.465341:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050094.465350:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050094.465367:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513405248/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050094.465383:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050094.465387:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050094.465416:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050094.465426:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405248:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050094.466788:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050094.466832:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:3.0:1781050094.466932:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:3.0:1781050094.466937:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050094.466939:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050094.466942:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405248:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050095.105281:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050095.361247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050096.129366:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050096.385337:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050097.153398:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050097.409192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050098.177369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050098.433325:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050099.201349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050099.457287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050099.585390:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050099.585396:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050099.585407:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050099.585419:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3735 00000100:00080000:3.0:1781050099.585422:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050099.585429:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050099.585438:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050099.585439:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513405312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050099.585453:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513405440/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050099.585495:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050099.585499:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050099.585527:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405440:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050099.585527:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513405376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050099.586807:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513405376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050099.586816:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513405312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050099.586829:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050099.586834:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050099.586838:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405440:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050100.225364:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050100.481287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050101.249321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050101.505282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050102.273354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050102.529292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050103.297306:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050103.553232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050104.321306:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050104.577222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050104.705377:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050104.705390:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050104.705396:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050104.705414:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3740 00000100:00100000:0.0:1781050104.705421:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050104.705421:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050104.705436:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050104.705452:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050104.705482:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a5a8ed39 x1867562513405632/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050104.705516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050104.705521:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050104.705569:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050104.705585:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405632:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050104.707348:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050104.707381:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050104.707409:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050104.707415:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050104.707423:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405632:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050105.345314:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050105.601268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050106.369316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050106.625262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050107.393300:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050107.649249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050108.417285:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050108.673276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050109.441301:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050109.697220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050109.825327:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050109.825334:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050109.825337:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050109.825349:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3745 00000100:00080000:1.0:1781050109.825352:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050109.825359:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781050109.825366:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405696:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781050109.825368:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050109.825384:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b12abf0d x1867562513405824/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050109.825423:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050109.825426:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050109.825450:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405760:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050109.825471:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405824:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050109.826895:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781050109.826897:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050109.826903:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050109.826909:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405760:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050109.826913:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513405824:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050110.465284:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050110.721257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050111.489360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050111.745226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050112.513276:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050112.769226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050113.537334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050113.793224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050114.561320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050114.817250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050114.945343:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050114.945343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050114.945354:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050114.945358:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050114.945364:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3750 00000100:00080000:1.0:1781050114.945367:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050114.945372:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050114.945382:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050114.945399:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513406016/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050114.945416:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050114.945418:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050114.945435:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405952:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050114.945464:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406016:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050114.946777:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513405952:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050114.946798:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513405888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050114.946805:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050114.946811:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050114.946818:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406016:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050115.585369:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050115.841282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050116.609334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050116.865196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050117.633346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050117.889956:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050118.657351:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050118.913201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050119.681361:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050119.937291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050120.065368:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050120.065377:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050120.065381:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050120.065396:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3755 00000100:00080000:3.0:1781050120.065400:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050120.065408:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050120.065418:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050120.065435:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513406208/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050120.065462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050120.065466:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050120.065496:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513406144:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050120.065591:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050120.065596:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513406080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050120.065699:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406208:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050120.068781:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513406080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050120.068818:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050120.068825:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050120.068831:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406208:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050120.068838:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513406144:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050120.705559:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050120.961285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050121.729432:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050121.985280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050122.753885:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050123.009267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050123.777381:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050124.035966:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050124.802762:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050125.057279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050125.185433:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050125.185442:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050125.185446:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050125.185458:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3761 00000100:00080000:1.0:1781050125.185461:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050125.185468:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050125.185479:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050125.185498:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513406400/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050125.185527:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050125.185530:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050125.185563:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050125.185667:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050125.185673:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513406272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050125.185759:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406400:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:3.0:1781050125.191504:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050125.191517:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050125.191528:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406400:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050125.192150:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513406272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050125.192169:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406336:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050125.827826:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050126.081545:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050126.849364:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050127.105266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050127.878844:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050128.129576:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050128.905149:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050129.155240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050129.921337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050130.178923:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050130.309061:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050130.309071:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050130.309075:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050130.309104:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3766 00000100:00080000:0.0:1781050130.309108:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050130.309114:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050130.309146:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050130.309166:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513406592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050130.309191:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050130.309195:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050130.315240:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050130.315465:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050130.315477:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513406464:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050130.321232:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406592:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050130.328509:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513406464:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050130.328531:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050130.328674:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050130.328680:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050130.328684:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406592:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050130.946893:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050131.201309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050131.969346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050132.225797:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050132.993353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050133.249225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050134.019753:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050134.273241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050135.041300:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050135.297246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050135.425379:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513406656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050135.425556:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050135.425561:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050135.425564:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050135.425569:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3771 00000100:00080000:1.0:1781050135.425571:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050135.425576:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050135.425587:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050135.425602:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001d23a43b x1867562513406784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050135.425654:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050135.425657:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050135.425673:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050135.425744:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050135.432426:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050135.434814:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050135.434826:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050135.434836:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050135.438436:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513406656:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050136.066646:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050136.321333:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050137.089363:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050137.345258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050138.113459:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050138.369281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050139.220322:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050139.393410:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050140.162069:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050140.421885:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050140.545438:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513406848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050140.549637:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050140.549653:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050140.549657:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050140.549670:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3776 00000100:00080000:1.0:1781050140.549698:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050140.549706:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050140.549720:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050140.549760:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513406976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050140.549798:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050140.549803:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050140.549848:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050140.550009:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406976:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050140.551629:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513406848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050140.552583:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050140.552591:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050140.552598:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513406976:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050140.553616:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513406912:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050141.193837:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050141.441303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050142.210553:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050142.465293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050143.233367:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050143.489291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050144.257367:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050144.513292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050145.281484:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050145.537515:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050145.665401:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050145.665412:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781050145.665541:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513407040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050145.665766:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050145.665772:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3781 00000100:00080000:0.0:1781050145.665775:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050145.665782:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050145.665793:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050145.665814:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513407168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050145.665969:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407168:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050145.666035:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050145.666041:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513407104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050145.666288:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050145.666293:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781050145.672168:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050145.672182:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050145.672195:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407168:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050145.672411:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513407104:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050145.672989:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513407040:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050146.305611:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050146.561296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050147.329510:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050147.585984:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050148.353947:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050148.609289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050149.377398:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050149.634172:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050150.401702:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050150.657837:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050150.786572:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050150.786579:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050150.786583:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050150.786593:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3786 00000100:00080000:0.0:1781050150.786597:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050150.786603:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050150.786614:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050150.786629:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513407360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781050150.786631:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513407232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050150.786732:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050150.786794:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050150.786800:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513407296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050150.788414:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050150.788419:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050150.796458:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513407232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050150.800176:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050150.800186:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050150.800195:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050150.800718:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513407296:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050151.425760:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050151.681459:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050152.449387:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050152.705252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050153.614523:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050153.730245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050154.625632:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050154.753397:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050155.649493:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050155.777312:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050155.905385:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050155.905395:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050155.905398:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050155.905411:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3791 00000100:00080000:2.0:1781050155.905415:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050155.905423:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050155.905433:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050155.905452:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084e7220c x1867562513407552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050155.905519:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407552:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050155.905625:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050155.905629:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050155.905649:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513407424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050155.905685:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050155.905690:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513407488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050155.911521:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513407488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050155.911543:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513407424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050155.911574:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050155.911581:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050155.911586:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407552:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050156.673463:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050156.801370:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050157.697346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050157.825257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050158.722150:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050158.849301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050159.745569:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050159.873881:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050160.769504:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050160.898948:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050161.025431:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050161.025439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050161.025442:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050161.025454:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3796 00000100:00080000:3.0:1781050161.025458:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050161.025465:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050161.025475:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050161.025493:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513407744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050161.025520:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050161.025523:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050161.025556:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513407616:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050161.025655:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050161.025661:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513407680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050161.025817:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407744:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050161.029270:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050161.029286:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050161.029297:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407744:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050161.030240:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513407680:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050161.030257:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513407616:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050161.793287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050161.921268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050162.817385:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050162.945245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050163.841422:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050163.969242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050164.865369:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050164.993256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050165.889424:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050166.017309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781050166.145423:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513407808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050166.145580:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050166.145587:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513407872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050166.145946:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050166.145952:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050166.145955:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050166.145961:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3802 00000100:00080000:3.0:1781050166.145964:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050166.145970:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050166.145982:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050166.146000:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513407936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050166.146080:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407936:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050166.146178:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050166.146182:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781050166.150499:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050166.150514:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050166.150526:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513407936:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050166.150634:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513407872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050166.150645:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513407808:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050166.913415:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050167.041350:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050167.938345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050168.065389:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050168.961426:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050169.089273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050169.985421:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050170.113293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050171.011161:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050171.137348:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050171.266267:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050171.266274:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050171.266278:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050171.266289:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3807 00000100:00080000:0.0:1781050171.266291:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050171.266311:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050171.266324:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050171.266342:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513408128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050171.266367:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050171.266371:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050171.266459:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513408000:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050171.266562:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050171.266569:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513408064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050171.266891:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408128:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050171.269752:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050171.269764:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050171.269773:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408128:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050171.269907:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513408064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050171.269919:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513408000:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050172.033470:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050172.161423:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050173.058307:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050173.185291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050174.081434:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050174.209282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050175.106232:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050175.233310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050176.131219:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050176.257264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050176.385364:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050176.385372:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050176.385376:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050176.385389:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3812 00000100:00080000:0.0:1781050176.385395:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050176.385401:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050176.385414:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050176.385435:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513408320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050176.385458:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050176.385462:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050176.385470:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513408256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050176.385507:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513408192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050176.385612:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408320:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050176.388946:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513408192:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050176.389315:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513408256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050176.389406:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050176.389412:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050176.389417:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408320:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050177.153777:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050177.281278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050178.177455:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050178.306975:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050179.201392:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050179.329632:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050180.225504:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050180.353249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050181.249584:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050181.377313:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050181.505469:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050181.505479:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050181.505482:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050181.505494:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3817 00000100:00080000:3.0:1781050181.505517:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050181.505524:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050181.505534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050181.505552:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513408512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050181.505620:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408512:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050181.505825:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513408384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050181.505867:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050181.505873:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513408448:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050181.506288:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050181.506294:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781050181.509387:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050181.509402:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050181.509414:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408512:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050181.509534:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513408448:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050181.509546:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513408384:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050182.275043:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050182.401261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050183.300100:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050183.426979:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050184.329686:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050184.449812:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050185.345372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050185.477543:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050186.369501:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050186.498951:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050186.625826:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050186.625836:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050186.625840:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050186.625853:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3822 00000100:00080000:2.0:1781050186.625857:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050186.625892:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050186.625902:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050186.625920:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000523c4981 x1867562513408704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050186.625945:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050186.625949:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050186.625982:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513408640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050186.626100:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050186.626106:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513408576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050186.627575:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408704:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050186.628353:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513408576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050186.636218:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050186.636235:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050186.636248:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408704:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050186.639940:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513408640:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050187.394418:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050187.521637:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050188.418040:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050188.545867:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050189.443870:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050189.569645:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050190.465447:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050190.593785:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050191.489538:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050191.617300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050191.751692:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050191.751704:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050191.751708:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050191.751722:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3827 00000100:00080000:2.0:1781050191.751726:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050191.751732:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050191.751744:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050191.751762:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513408896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050191.751820:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050191.751825:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050191.751859:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513408832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050191.751998:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050191.752005:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513408768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050191.752150:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408896:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050191.765421:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513408768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050191.765452:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513408832:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050191.765764:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050191.765770:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050191.765775:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513408896:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050192.519824:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050192.644764:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050193.537419:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050193.665296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050194.562693:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050194.689330:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050195.593024:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050195.713936:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050196.609377:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050196.737387:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781050196.867353:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513408960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050196.867548:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050196.867555:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513409024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050196.868060:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050196.868066:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050196.868070:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050196.868077:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3832 00000100:00080000:3.0:1781050196.868079:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050196.868085:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050196.868096:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050196.868114:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513409088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050196.868186:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409088:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050196.868297:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050196.868302:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781050196.893975:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050196.893989:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050196.894000:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409088:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050196.896903:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513409024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050196.896941:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513408960:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050197.633949:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050197.761842:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050198.661155:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050198.786377:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050199.683947:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050199.809301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050200.710857:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050200.833953:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050201.730822:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050201.857768:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050201.986597:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050201.986605:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050201.986608:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050201.986620:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3837 00000100:00080000:0.0:1781050201.986624:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050201.986630:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050201.986640:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050201.986657:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513409280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050201.986680:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050201.986683:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050201.986779:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513409152:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050201.986883:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050201.986889:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513409216:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050201.987038:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409280:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781050201.989626:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050201.989637:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050201.989647:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409280:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050201.990991:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513409152:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050201.999404:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513409216:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050202.757120:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050202.891093:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050203.782867:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050203.905876:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050204.808191:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050204.929262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050205.826669:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050205.954303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050206.849418:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050206.977434:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050207.107928:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050207.107936:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050207.107940:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050207.107950:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3843 00000100:00080000:3.0:1781050207.107954:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050207.107960:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050207.107971:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050207.107985:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513409472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050207.108005:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050207.108009:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050207.108154:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409472:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050207.110506:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513409408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050207.110599:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050207.110607:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513409344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050207.115941:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050207.115952:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050207.115961:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409472:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050207.123641:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513409344:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050207.123666:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513409408:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050207.873907:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050208.004074:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050208.897597:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050209.025851:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050209.921443:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050210.049516:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050210.945547:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050211.073297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050211.970889:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050212.097246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050212.228340:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050212.228349:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050212.228353:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050212.228366:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3848 00000100:00080000:3.0:1781050212.228371:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050212.228378:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050212.228392:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050212.228419:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513409664/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050212.228452:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050212.228455:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050212.229399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513409536:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050212.229544:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050212.229552:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513409600:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050212.231695:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409664:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050212.238787:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050212.238800:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050212.238810:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409664:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050212.266887:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513409536:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050212.272973:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513409600:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050212.993385:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050213.121300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050214.017837:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050214.146572:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050215.041724:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050215.169784:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050216.077386:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050216.200329:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050217.090228:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050217.217611:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050217.346011:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050217.346020:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050217.346024:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050217.346054:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3853 00000100:00080000:0.0:1781050217.346058:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050217.346099:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050217.346110:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050217.346170:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513409856/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050217.346200:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050217.346204:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050217.346304:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513409728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050217.346461:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050217.346468:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513409792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050217.346722:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409856:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050217.350622:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513409792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050217.353979:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050217.353993:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050217.354004:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513409856:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050217.358207:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513409728:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050218.115546:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050218.242029:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050219.138707:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050219.269450:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050220.164316:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050220.289317:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050221.187666:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050221.313298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050222.209760:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050222.337423:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781050222.466848:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513409920:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050222.467006:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050222.470409:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513409984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050222.473622:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050222.473682:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050222.473686:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050222.473697:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3858 00000100:00080000:3.0:1781050222.473701:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050222.473707:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050222.473720:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050222.473753:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bc445f2a x1867562513410048/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050222.473787:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050222.473791:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050222.474002:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410048:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050222.479505:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050222.479519:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050222.479530:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410048:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050222.484239:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513409920:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050222.487727:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513409984:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050223.233546:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050223.361364:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050224.259161:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050224.385404:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050225.281436:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050225.411299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050226.305569:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050226.433416:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050227.335432:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050227.459255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050227.589327:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050227.589338:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050227.589341:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050227.589354:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3863 00000100:00080000:1.0:1781050227.589357:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050227.589366:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050227.589377:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050227.589397:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513410240/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050227.589426:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050227.589429:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050227.589462:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050227.589571:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050227.589578:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513410112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050227.590442:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410240:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050227.592645:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050227.593616:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513410112:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050227.595356:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050227.595369:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050227.595378:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410240:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050228.354071:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050228.481521:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050229.393393:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050229.507279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050230.403620:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050230.530354:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050231.425565:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050231.553681:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050232.455470:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050232.577322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050232.706411:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050232.706438:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050232.706441:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050232.706453:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3868 00000100:00080000:1.0:1781050232.706456:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050232.706463:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050232.706473:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050232.706494:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c26c4b8c x1867562513410432/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050232.706764:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050232.706768:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050232.706798:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050232.707071:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050232.707179:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513410304:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050232.707214:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410432:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050232.714235:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050232.714251:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050232.714264:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410432:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050232.714443:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513410304:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050232.720117:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410368:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050233.479329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050233.602256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050234.502390:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050234.626560:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050235.523737:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050235.652465:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050236.549323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050236.673306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050237.569457:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050237.697273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050237.830850:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050237.830882:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050237.830886:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050237.830899:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3873 00000100:00080000:3.0:1781050237.830902:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050237.830910:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050237.830922:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050237.830941:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513410624/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050237.830963:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050237.830966:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050237.830996:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513410560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050237.831102:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050237.831108:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513410496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050237.831504:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410624:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050237.835398:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513410496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050237.835429:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513410560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050237.840952:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050237.840968:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050237.840998:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410624:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050238.595287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050238.721314:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050239.617618:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050239.747175:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050240.641521:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050240.769215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050241.665725:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050241.793299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050242.689872:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050242.818217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050242.946053:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050242.946063:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050242.946067:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050242.946081:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3878 00000100:00080000:1.0:1781050242.946084:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050242.946092:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781050242.946096:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513410688:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:1.0:1781050242.946103:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050242.946121:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084471047 x1867562513410816/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050242.946180:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050242.946183:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050242.946207:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050242.946285:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410816:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050242.977629:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513410688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050242.977761:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050242.977766:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050242.977771:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513410816:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050242.978783:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410752:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050243.713853:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050243.842117:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050244.737459:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050244.865807:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050245.761425:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050245.889374:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050245.953373:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050245.953382:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050245.953386:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050245.953399:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3883 00000100:00080000:1.0:1781050245.953403:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050245.953410:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050245.953438:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050245.953456:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513411008/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050245.953480:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050245.953484:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050245.953516:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050245.953623:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050245.953629:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513410880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050245.954356:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:02020000:2.0:1781050245.963028:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:2.0:1781050245.963045:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050245.963051:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050245.963056:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050245.963801:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513410880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050245.963824:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513410944:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050246.785592:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050246.913310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050247.809526:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050247.937363:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050248.833415:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050248.961326:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050249.860175:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050249.985285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050250.882053:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050251.009503:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050251.137405:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050251.137414:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050251.137417:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050251.137433:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3887 00000100:00080000:1.0:1781050251.137436:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050251.137444:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050251.137454:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050251.137472:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513411200/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050251.137497:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050251.137501:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050251.137616:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513411136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050251.137618:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411200:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050251.137849:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513411072:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050251.153245:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513411136:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050251.153907:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050251.153915:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050251.153920:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411200:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050251.156766:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513411072:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050251.909245:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050252.033286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050252.935785:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050253.057350:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050253.955199:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050254.082564:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050254.977598:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050255.105311:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050256.001951:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050256.130340:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050256.257451:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050256.257461:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050256.257495:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050256.257507:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3892 00000100:00080000:3.0:1781050256.257511:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050256.257518:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050256.257529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050256.257531:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513411264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050256.257549:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513411392/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050256.257577:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050256.257581:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050256.257610:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513411328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050256.257762:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050256.271923:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513411264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050256.272171:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050256.272177:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050256.272182:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050256.272264:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513411328:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050257.025753:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050257.153326:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050258.049497:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050258.177738:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050259.073715:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050259.201286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050260.098500:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050260.225247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050261.121945:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050261.249277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050261.377450:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050261.377460:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050261.377463:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050261.377476:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3897 00000100:00080000:1.0:1781050261.377479:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050261.377486:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050261.377496:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050261.377511:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b12abf0d x1867562513411584/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050261.377532:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050261.377535:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050261.377570:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513411520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050261.377572:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411584:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050261.377691:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513411456:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050261.397866:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050261.397881:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050261.397892:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411584:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050261.397944:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513411520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050261.398001:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513411456:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050262.146468:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050262.273286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050263.169402:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050263.297284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050264.193346:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050264.321314:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050265.218475:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050265.345299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050266.241655:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050266.369430:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050266.502780:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050266.502792:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050266.502796:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050266.502814:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3902 00000100:00080000:1.0:1781050266.502818:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050266.502827:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050266.502840:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050266.502861:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b738af86 x1867562513411776/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050266.502900:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050266.502903:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050266.502937:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513411712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050266.502968:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050266.503047:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050266.503057:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513411648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050266.505279:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050266.505302:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050266.505312:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050266.506393:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513411648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050266.506697:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513411712:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050267.265371:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050267.393248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050268.292804:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050268.417294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050269.314645:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050269.441294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050270.338422:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050270.465325:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050271.362418:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050271.489289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050271.617444:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050271.617453:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050271.617457:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050271.617470:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3907 00000100:00080000:3.0:1781050271.617474:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050271.617481:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050271.617494:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050271.617523:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513411968/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050271.617566:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050271.617570:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050271.617613:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513411904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050271.617732:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050271.617739:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513411840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050271.617963:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411968:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050271.624971:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513411904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050271.625105:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513411840:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050271.630743:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050271.630764:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050271.630778:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513411968:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050272.389780:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050272.513245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050273.409907:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050273.537416:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050274.440772:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050274.561658:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050275.458037:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050275.585351:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050276.482313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050276.609451:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050276.738074:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050276.738083:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050276.738087:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050276.738099:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3912 00000100:00080000:3.0:1781050276.738103:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050276.738164:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050276.738174:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050276.738192:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513412160/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050276.738217:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050276.738220:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050276.738291:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513412032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050276.738408:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050276.738416:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513412096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050276.739177:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412160:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1781050276.741918:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050276.741932:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050276.741942:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412160:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050276.742035:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513412096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050276.744999:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513412032:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050277.505552:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050277.635709:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050278.529702:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050278.657292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050279.553369:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050279.681253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050280.577771:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050280.705289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050281.602528:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050281.729278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050281.859758:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050281.859766:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050281.859770:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050281.859781:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3917 00000100:00080000:1.0:1781050281.859784:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050281.859939:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050281.859959:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050281.860014:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b738af86 x1867562513412352/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050281.860035:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513412224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050281.860186:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412352:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050281.860224:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050281.860237:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513412288:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050281.860883:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050281.860888:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050281.864507:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513412288:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050281.866469:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513412224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050281.867576:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050281.867587:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050281.867604:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412352:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050282.631552:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050282.753313:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050283.649433:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050283.777275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050284.675534:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050284.801323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050285.698615:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050285.825397:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050286.721546:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050286.849336:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050286.977917:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050286.977933:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050286.977936:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050286.977948:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3922 00000100:00080000:2.0:1781050286.977952:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050286.977959:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050286.977970:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050286.977988:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513412544/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050286.978010:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050286.978034:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050286.978067:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513412480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050286.978190:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050286.978197:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513412416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050286.978484:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412544:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050286.985099:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513412416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050286.985179:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513412480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050286.985278:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050286.985283:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050286.985288:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412544:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050287.745297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050287.873364:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050288.769484:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050288.897317:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050289.793484:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050289.921271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050290.817727:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050290.946289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050291.843669:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050291.970212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050292.099008:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050292.099016:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050292.099020:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050292.099032:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3928 00000100:00080000:1.0:1781050292.099036:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050292.099042:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050292.099053:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050292.099078:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513412736/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050292.099107:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050292.099110:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050292.099169:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513412608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050292.099233:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513412672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050292.101185:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412736:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050292.103471:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513412672:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050292.105424:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513412608:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050292.112113:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050292.112156:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050292.112167:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412736:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781050292.865421:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050292.993215:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050293.890282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050294.017288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050294.914602:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050295.041260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050295.937449:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050296.065321:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050296.963359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050297.089778:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050297.217927:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050297.217934:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050297.217938:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050297.217949:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3933 00000100:00080000:1.0:1781050297.217953:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050297.217959:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050297.217968:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050297.217983:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001cbc13b4 x1867562513412928/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050297.218006:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050297.218009:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050297.218374:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513412864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050297.218624:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050297.218633:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513412800:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050297.219618:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412928:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050297.221245:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513412800:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050297.228786:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513412864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050297.231667:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050297.231682:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050297.231694:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513412928:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050297.986814:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050298.115952:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050299.009878:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050299.137292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050300.033828:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050300.161321:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050301.057527:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050301.185492:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050302.081373:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050302.209308:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050302.337360:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050302.337383:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513412992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050302.337496:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050302.337501:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050302.337505:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050302.337509:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3938 00000100:00080000:1.0:1781050302.337512:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050302.337516:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050302.337527:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050302.337540:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008bfccf5d x1867562513413120/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050302.337563:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050302.337565:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050302.337578:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513413056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050302.338224:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413120:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050302.344622:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513413056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050302.344649:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513412992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050302.344671:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050302.344683:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050302.344694:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413120:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050303.106104:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050303.235162:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050304.130284:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050304.257308:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050305.154436:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050305.281267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050306.178775:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050306.305286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050307.215363:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050307.329432:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050307.459011:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050307.459029:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050307.459033:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050307.459043:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3943 00000100:00100000:0.0:1781050307.459047:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513413184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050307.459048:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050307.459081:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050307.459097:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050307.459117:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513413312/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050307.459167:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050307.459170:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050307.459200:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413312:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050307.459205:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513413248:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050307.463048:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513413184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050307.464333:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513413248:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050307.464549:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050307.464555:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050307.464560:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413312:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781050308.225424:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050308.353306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050309.255260:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050309.377267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050310.273373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050310.401253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050311.298424:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050311.425362:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050312.321449:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050312.449291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050312.578964:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050312.578973:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050312.578976:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050312.578989:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3948 00000100:00080000:3.0:1781050312.578992:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050312.578998:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050312.579009:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050312.579025:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513413504/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050312.579051:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050312.579055:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050312.579087:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513413440:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050312.579277:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050312.579285:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513413376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050312.579847:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413504:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050312.586222:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513413440:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050312.586266:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513413376:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050312.588321:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050312.588335:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050312.588345:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413504:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050313.345541:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050313.473286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050314.373487:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050314.498202:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050315.393615:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050315.523858:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050316.422093:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050316.545348:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050317.442750:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050317.569298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050317.699059:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050317.699068:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050317.699071:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050317.699084:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3953 00000100:00080000:0.0:1781050317.699087:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050317.699094:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050317.699105:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050317.699120:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513413696/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050317.699209:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050317.699212:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050317.699350:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513413568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050317.699485:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050317.699492:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513413632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050317.701427:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413696:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050317.701561:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513413632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050317.702401:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513413568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050317.703747:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050317.703759:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050317.703947:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413696:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050318.466946:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050318.593272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050319.489565:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050319.618420:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050320.515750:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050320.642482:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050321.537512:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050321.665662:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050322.561347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050322.689347:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050322.817471:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050322.817481:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050322.817484:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050322.817496:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3958 00000100:00080000:1.0:1781050322.817499:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050322.817506:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050322.817516:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050322.817533:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000000dbb310 x1867562513413888/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050322.817559:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050322.817563:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050322.817597:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513413824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050322.817656:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413888:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050322.817711:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050322.817722:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513413760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050322.822494:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050322.822510:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050322.822524:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513413888:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050322.822710:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513413760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050322.822730:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513413824:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050323.588952:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050323.713517:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050324.611257:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050324.737357:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050325.633505:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050325.761360:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050326.662881:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050326.785522:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050327.681413:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050327.809381:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050327.937463:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050327.937472:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050327.937475:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050327.937489:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3963 00000100:00080000:3.0:1781050327.937492:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050327.937511:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050327.937522:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050327.937551:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513414080/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050327.937576:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513413952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050327.937583:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050327.937587:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050327.937637:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513414016:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050327.938600:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414080:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050327.941465:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513413952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050327.950308:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513414016:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050327.955061:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050327.955076:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050327.955088:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414080:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050328.706659:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050328.833707:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050329.730452:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050329.858420:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050330.753457:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050330.881346:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050331.782839:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050331.905463:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050332.802360:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050332.929288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050333.057598:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050333.057606:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050333.057610:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050333.057642:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3968 00000100:00080000:1.0:1781050333.057646:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050333.057652:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050333.057662:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050333.057679:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ea0bf1d5 x1867562513414272/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050333.057709:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050333.057712:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050333.057742:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513414208:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050333.057840:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050333.057846:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513414144:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050333.057948:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414272:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050333.060533:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513414144:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050333.060961:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050333.060967:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050333.060971:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414272:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050333.061837:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513414208:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050333.825380:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050333.955892:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050334.853068:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050334.977273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050335.879400:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050336.001450:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050336.897723:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050337.025327:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050337.921569:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050338.049298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050338.179595:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513414336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050338.179860:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050338.179866:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050338.179869:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050338.179875:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3974 00000100:00100000:1.0:1781050338.180271:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050338.180279:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513414400:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050338.180362:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050338.180369:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050338.180379:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050338.180393:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018617087 x1867562513414464/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050338.180414:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050338.180417:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050338.180479:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414464:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050338.184601:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050338.184615:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050338.184627:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414464:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050338.187819:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513414400:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050338.187843:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513414336:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050338.946641:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050339.073376:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050339.976094:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050340.097293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050340.993465:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050341.121242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050342.017547:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050342.145297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050343.044036:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050343.169526:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050343.297475:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050343.297484:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050343.297487:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050343.297499:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3979 00000100:00080000:3.0:1781050343.297503:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050343.297509:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050343.297520:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050343.297550:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b651757 x1867562513414656/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050343.297583:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050343.297587:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050343.297617:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513414592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050343.297709:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050343.297716:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513414528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050343.298036:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414656:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050343.302764:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513414528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050343.302799:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513414592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050343.302861:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050343.302872:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050343.302882:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414656:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050344.066003:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050344.193267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050345.090782:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050345.217298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050346.113357:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050346.241290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050347.137424:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050347.266904:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050348.161937:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050348.289334:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050348.417603:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050348.417613:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050348.417618:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050348.417631:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3984 00000100:00080000:3.0:1781050348.417635:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050348.417642:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050348.417655:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050348.417673:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513414848/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050348.417713:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050348.417717:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050348.417823:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414848:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050348.418522:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513414720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050348.418587:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050348.418595:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513414784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050348.420718:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513414784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050348.420742:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513414720:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050348.424065:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050348.424077:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050348.424088:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513414848:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050349.185503:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050349.313262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050350.209513:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050350.337249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050351.233331:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050351.361686:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050352.258600:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050352.385293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050353.281360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050353.409580:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050353.539620:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050353.539628:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050353.539631:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050353.539644:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3989 00000100:00080000:3.0:1781050353.539648:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050353.539656:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050353.539667:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050353.539683:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000087a1080b x1867562513415040/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050353.539709:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050353.539712:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050353.539765:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513414976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050353.539823:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513414912:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050353.541436:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415040:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781050353.543936:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050353.543950:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050353.543962:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415040:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050353.546326:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513414976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050353.548724:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513414912:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050354.306114:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050354.433332:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050355.329387:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050355.457275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050356.354475:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050356.481369:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050357.377388:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050357.505249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050358.402196:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050358.529442:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050358.660043:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050358.660052:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050358.660056:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050358.660088:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3994 00000100:00080000:3.0:1781050358.660092:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050358.660098:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050358.660111:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050358.660187:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513415104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050358.660350:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050358.660359:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513415168:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050358.660495:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513415232/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050358.660514:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050358.660517:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050358.660999:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415232:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050358.666061:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513415168:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050358.669675:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513415104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050358.672735:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050358.672751:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050358.672763:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415232:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050359.428181:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050359.553241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050360.451460:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050360.577285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050361.474667:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050361.601217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050362.499291:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050362.628775:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050363.522832:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050363.649270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050363.777708:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050363.777715:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050363.777718:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050363.777730:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 3999 00000100:00080000:1.0:1781050363.777733:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050363.777739:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050363.777749:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050363.777763:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a161b523 x1867562513415424/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050363.777787:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050363.777790:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050363.777820:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513415360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050363.777920:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050363.777926:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513415296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050363.778254:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415424:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050363.781296:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513415296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050363.781322:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513415360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050363.789431:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050363.789444:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050363.789456:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415424:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050364.545388:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050364.673325:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050365.569523:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050365.697292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050366.593354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050366.721717:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050367.619152:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050367.745265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050368.646440:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050368.770097:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050368.897411:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050368.897420:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050368.897424:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050368.897436:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4004 00000100:00080000:1.0:1781050368.897439:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050368.897446:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050368.897459:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050368.897487:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513415616/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050368.897512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050368.897516:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050368.897547:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513415552:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050368.897672:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050368.897678:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513415488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050368.897783:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415616:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050368.900078:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050368.900090:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050368.900102:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415616:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050368.902723:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513415488:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050368.902752:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513415552:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050369.665505:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050369.793272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050370.689499:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050370.817253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050371.713422:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050371.843185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050372.748629:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050372.868185:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050373.761378:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050373.889343:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050374.017453:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050374.017475:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050374.017480:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050374.017495:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4009 00000100:00080000:3.0:1781050374.017499:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050374.017508:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050374.017522:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050374.017531:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513415680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050374.017547:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513415808/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050374.017575:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050374.017596:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050374.017652:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513415744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050374.018343:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415808:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050374.021882:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513415744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050374.022016:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050374.022026:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050374.022034:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513415808:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050374.022254:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513415680:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050374.789114:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050374.913354:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050375.809858:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050375.937264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050376.833545:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050376.961354:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050377.857495:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050377.986815:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050378.881449:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050379.009538:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781050379.137927:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513415872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050379.138450:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050379.138458:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513415936:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050379.139554:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050379.139563:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050379.139567:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050379.139576:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4015 00000100:00080000:3.0:1781050379.139580:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050379.139655:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050379.139668:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050379.139692:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513416000/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050379.141857:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416000:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050379.145863:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050379.145878:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050379.147622:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513415936:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050379.147644:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513415872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050379.151616:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050379.151631:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050379.151641:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416000:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050379.905318:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050380.033313:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050380.931712:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050381.057343:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050381.956730:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050382.081308:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050382.977412:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050383.105328:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050384.002314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050384.129703:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050384.258920:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513416064:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050384.260293:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050384.260310:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513416128:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050384.260884:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050384.260908:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050384.260912:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050384.260918:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4020 00000100:00080000:1.0:1781050384.260921:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050384.260927:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050384.260938:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050384.260960:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000013bcd30c x1867562513416192/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050384.260986:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050384.260989:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050384.261038:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416192:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050384.270505:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513416128:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050384.270527:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513416064:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050384.275434:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050384.275449:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050384.275459:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416192:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050385.027378:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050385.153407:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050386.049415:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050386.177925:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050387.074567:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050387.202309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050388.097420:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050388.225514:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050389.125784:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050389.249300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050389.377922:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513416256:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050389.378240:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050389.378264:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050389.378267:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050389.378273:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4025 00000100:00080000:0.0:1781050389.378276:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050389.378283:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050389.378294:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050389.378312:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513416384/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050389.378341:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050389.378344:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050389.378370:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513416320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050389.378932:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416384:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050389.383191:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050389.383206:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050389.383218:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416384:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050389.385515:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513416256:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050389.389691:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513416320:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050390.149618:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050390.273318:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050391.172308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050391.297362:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050392.193459:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050392.321288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050393.218032:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050393.346148:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050394.241429:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050394.369904:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050394.498466:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050394.498475:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050394.498479:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050394.498492:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4030 00000100:00080000:3.0:1781050394.498495:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050394.498502:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050394.498513:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050394.498529:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513416576/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050394.498552:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050394.498556:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050394.498659:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513416448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050394.498766:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050394.498772:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513416512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050394.501039:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416576:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050394.503874:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513416512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050394.503897:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513416448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050394.510431:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050394.510446:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050394.510459:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416576:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050395.265410:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050395.393384:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050396.289528:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050396.417292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050397.313366:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050397.441282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050398.338678:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050398.465381:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050399.362474:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050399.489289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050399.617471:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050399.617478:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050399.617481:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050399.617494:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4035 00000100:00080000:3.0:1781050399.617498:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050399.617504:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050399.617515:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050399.617531:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d139f817 x1867562513416768/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050399.617575:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050399.617578:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050399.617719:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416768:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050399.618086:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513416704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050399.618172:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050399.618178:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513416640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050399.629571:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513416640:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050399.629598:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513416704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:0.0:1781050399.656518:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781050399.656532:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050399.656538:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050399.656542:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416768:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781050400.386794:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050400.513293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050401.409440:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050401.539565:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050402.433705:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050402.562063:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050403.457456:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050403.585749:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050404.481654:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050404.609293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050404.738054:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050404.738064:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050404.738068:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050404.738081:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4040 00000100:00080000:1.0:1781050404.738085:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050404.738092:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050404.738122:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050404.738199:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000013bcd30c x1867562513416960/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050404.738220:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050404.738224:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050404.738248:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513416832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050404.738308:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513416896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050404.738708:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416960:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050404.742371:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513416896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050404.742613:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513416832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050404.744097:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050404.744107:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050404.744116:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513416960:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050405.506453:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050405.634363:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050406.529413:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050406.657510:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050407.553709:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050407.681302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050408.577394:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050408.705300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050409.604161:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050409.730171:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050409.857428:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050409.857437:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050409.857441:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050409.857453:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4045 00000100:00080000:0.0:1781050409.857456:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050409.857463:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050409.857501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050409.857521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513417152/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050409.857547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050409.857551:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050409.857636:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513417024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050409.857762:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050409.857769:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513417088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050409.857880:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417152:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050409.862973:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513417088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050409.862991:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513417024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050409.863232:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050409.863238:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050409.863244:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417152:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050410.630113:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050410.753323:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050411.650448:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050411.777439:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050412.673423:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050412.807254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050413.697792:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050413.825347:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050414.721524:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050414.850084:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050414.981063:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513417216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050414.981368:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050414.981374:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050414.981378:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050414.981385:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4050 00000100:00080000:1.0:1781050414.981388:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050414.981407:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050414.981435:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050414.981455:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d7e018b1 x1867562513417344/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050414.981476:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050414.981479:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050414.981503:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513417280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050414.981559:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417344:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050414.986931:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513417280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050414.989998:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513417216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050414.995798:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050414.995813:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050414.995834:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417344:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050415.746637:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050415.873338:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050416.773010:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050416.897300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050417.793977:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050417.924388:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050418.819509:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050418.945332:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050419.841458:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050419.969519:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050420.097861:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050420.097870:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050420.097873:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050420.097886:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4056 00000100:00080000:1.0:1781050420.097890:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050420.097897:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050420.097915:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050420.097933:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513417536/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050420.097962:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050420.097983:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050420.098017:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513417472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050420.098146:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050420.098154:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513417408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050420.099955:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417536:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050420.103261:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513417408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050420.103589:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513417472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050420.104646:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050420.104732:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050420.104742:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417536:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050420.867543:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050420.993291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050421.889491:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050422.017291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050422.917608:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050423.041289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050423.937464:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050424.065279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050424.961413:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050425.089790:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050425.217520:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050425.217530:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050425.217533:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050425.217545:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4061 00000100:00080000:3.0:1781050425.217549:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050425.217555:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050425.217565:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050425.217580:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513417728/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050425.217604:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050425.217607:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050425.217654:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513417664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050425.217743:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050425.217762:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513417600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050425.218037:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417728:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1781050425.224732:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050425.224747:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050425.224759:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417728:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050425.225183:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513417600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050425.225224:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513417664:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050425.987107:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050426.113294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050427.009441:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050427.137310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050428.036749:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050428.161290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050429.059575:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050429.188226:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050430.082393:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050430.209284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050430.337388:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050430.337406:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050430.337410:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050430.337421:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4066 00000100:00080000:1.0:1781050430.337425:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050430.337431:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050430.337444:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050430.337461:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a161b523 x1867562513417920/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050430.337499:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050430.337502:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050430.337534:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513417856:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050430.337587:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417920:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050430.337656:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050430.337665:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513417792:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050430.347591:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050430.347604:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050430.347625:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513417920:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050430.348847:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513417792:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050430.350002:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513417856:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050431.109611:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050431.233309:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050432.129469:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050432.257422:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050433.155838:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050433.282095:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050434.177254:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050434.306083:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050435.201448:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050435.329255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050435.457420:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050435.457438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050435.457442:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050435.457455:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4071 00000100:00080000:1.0:1781050435.457458:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050435.457465:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050435.457478:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050435.457496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ea0bf1d5 x1867562513418112/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050435.457513:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050435.457516:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050435.457550:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513418048:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050435.457675:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050435.457680:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513417984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050435.458050:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418112:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050435.465643:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050435.465657:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050435.465668:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418112:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050435.466585:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513417984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050435.466605:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513418048:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050436.225497:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050436.353260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050437.249661:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050437.377672:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050438.274034:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050438.402526:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050439.297551:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050439.425241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050440.323258:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050440.449286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050440.577444:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050440.577453:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050440.577458:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050440.577469:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4076 00000100:00080000:2.0:1781050440.577473:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050440.577480:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050440.577494:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050440.577526:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001d6480a x1867562513418304/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050440.577550:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050440.577554:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050440.577583:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513418240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050440.577688:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050440.577695:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513418176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050440.577971:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418304:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050440.580882:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513418176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050440.580906:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513418240:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050440.583867:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050440.583880:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050440.583890:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418304:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050441.345813:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050441.473303:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050442.369902:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050442.497408:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050443.393446:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050443.521301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050444.417800:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050444.545322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050445.441324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050445.569285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050445.697383:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050445.697391:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050445.697395:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050445.697406:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4081 00000100:00080000:3.0:1781050445.697409:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050445.697416:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050445.697427:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050445.697444:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ed3dd285 x1867562513418496/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050445.697472:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050445.697475:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050445.697541:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418496:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050445.697796:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513418432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050445.697832:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050445.697836:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513418368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050445.700944:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513418368:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050445.700969:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513418432:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050445.702227:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050445.702235:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050445.702244:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418496:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781050446.469399:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050446.593467:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050447.490742:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050447.617996:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050448.513859:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050448.641926:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050449.541717:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050449.668254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050450.561683:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050450.689492:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050450.823715:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050450.823724:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050450.823728:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050450.823742:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4086 00000100:00080000:1.0:1781050450.823746:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050450.823773:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050450.823786:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050450.823805:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000000dbb310 x1867562513418688/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050450.823836:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050450.823840:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050450.823876:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513418624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050450.823893:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418688:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050450.823985:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513418560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050450.828613:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050450.828630:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050450.828641:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418688:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050450.831449:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513418560:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050450.835454:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513418624:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050451.586417:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050451.715193:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050452.611706:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050452.737344:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050453.634924:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050453.761334:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050454.659789:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050454.786527:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050455.681657:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050455.809343:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050455.937573:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513418752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050455.938051:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050455.938061:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513418816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050455.939860:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050455.939871:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050455.939875:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050455.939884:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4091 00000100:00080000:0.0:1781050455.939887:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 6s 00000100:00080000:0.0:1781050455.940042:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050455.940054:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050455.940078:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513418880/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781050455.940145:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418880:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050455.940223:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050455.940227:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050455.942446:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513418752:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050455.942489:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513418816:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050455.946958:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050455.946992:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050455.947003:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513418880:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050456.705991:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050456.833483:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050457.729421:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050457.857292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050458.753346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050458.886301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050459.777757:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050459.905277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050460.801321:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050460.929288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050461.057625:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050461.057633:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050461.057636:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050461.057648:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4096 00000100:00080000:2.0:1781050461.057652:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 11s 00000100:00080000:2.0:1781050461.057659:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050461.057669:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050461.057686:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513419072/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050461.057713:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050461.057717:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050461.057738:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513419008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050461.058038:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050461.058063:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513418944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050461.058175:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050461.060615:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513419008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050461.063956:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050461.063971:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050461.063982:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050461.070279:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513418944:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050461.825389:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050461.953294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050462.852401:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050462.977335:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050463.874267:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050464.001261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050464.901458:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050465.026045:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050465.922488:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050466.049658:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050466.177418:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050466.177428:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050466.177432:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050466.177444:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4102 00000100:00080000:1.0:1781050466.177448:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00080000:1.0:1781050466.177454:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050466.177467:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050466.177484:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000129aa103 x1867562513419264/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050466.177505:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050466.177509:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050466.177542:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513419200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050466.177725:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419264:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050466.177816:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513419136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050466.187398:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050466.187411:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050466.187423:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419264:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050466.190506:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513419200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050466.191895:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513419136:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050466.945558:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050467.073329:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050467.969411:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050468.097234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050468.993355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050469.121204:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050470.019782:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050470.145244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050471.042325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050471.169934:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050471.299995:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050471.300005:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050471.300010:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050471.300022:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4107 00000100:00080000:0.0:1781050471.300027:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050471.300034:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050471.300045:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050471.300064:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513419456/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050471.300087:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050471.300108:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050471.300264:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513419328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050471.300389:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050471.300396:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513419392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050471.300974:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419456:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050471.301993:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513419392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050471.313861:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050471.313875:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050471.313886:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419456:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050471.315018:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513419328:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050472.065312:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050472.195864:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050473.089890:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050473.217802:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050474.113355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050474.241382:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050475.137374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050475.265250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050476.161807:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050476.289298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050476.417322:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513419520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050476.418316:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050476.418324:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050476.418327:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050476.418337:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4112 00000100:00080000:0.0:1781050476.418340:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050476.418347:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781050476.418356:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000008:00100000:0.0:1781050476.418357:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050476.418370:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513419584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050476.418395:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d26e1cb9 x1867562513419648/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050476.418425:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050476.418428:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050476.418476:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419648:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050476.423471:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513419584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050476.424545:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050476.424554:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050476.424562:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419648:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050476.429353:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513419520:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050477.185320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050477.313252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050478.209310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050478.337278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050479.233419:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050479.363567:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050480.257343:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050480.385589:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050481.281819:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050481.409283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050481.537379:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050481.537386:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050481.537390:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050481.537402:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4117 00000100:00080000:0.0:1781050481.537406:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050481.537414:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050481.537424:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050481.537441:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513419840/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050481.537508:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050481.537512:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050481.538026:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419840:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050481.539764:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513419776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050481.539833:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050481.539840:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513419712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050481.543397:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513419776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050481.544180:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513419712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050481.547017:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050481.547030:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050481.547040:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513419840:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050482.305385:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050482.433301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050483.329364:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050483.457279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050484.353491:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050484.481375:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050485.377395:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050485.505276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050486.401334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050486.529242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050486.657453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050486.657463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050486.657467:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050486.657478:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4122 00000100:00080000:1.0:1781050486.657482:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050486.657488:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050486.657501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050486.657517:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008bfccf5d x1867562513420032/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050486.657586:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420032:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1781050486.657708:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050486.657713:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050486.657731:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513419968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050486.657771:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050486.657777:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513419904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050486.659732:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513419904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050486.662332:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513419968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050486.666485:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050486.666495:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050486.666504:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420032:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050487.425435:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050487.553417:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050488.449572:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050488.577264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050489.474594:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050489.601239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050490.499400:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050490.626511:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050491.521445:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050491.649285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050491.778096:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513420096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050491.779255:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050491.779284:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513420160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050491.779758:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050491.779764:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050491.779767:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050491.779772:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4127 00000100:00080000:0.0:1781050491.779775:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050491.779782:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050491.779793:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050491.779812:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513420224/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050491.779866:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420224:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050491.779919:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050491.779923:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050491.781106:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513420160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050491.781932:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513420096:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050491.786151:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050491.786163:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050491.786174:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420224:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050492.546247:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050492.673298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050493.569567:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050493.697483:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050494.593343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050494.723365:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050495.617472:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050495.745299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050496.643389:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050496.769291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050496.897477:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513420288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050496.897815:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050496.897823:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513420352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050496.898275:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050496.898281:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050496.898284:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050496.898290:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4132 00000100:00080000:2.0:1781050496.898292:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781050496.898298:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050496.898308:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050496.898901:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000015c9d1f3 x1867562513420416/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781050496.898955:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050496.899016:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050496.899019:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050496.899761:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513420352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050496.899774:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513420288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050496.900556:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050496.900562:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050496.900568:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420416:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050497.665353:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050497.793327:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050498.691664:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050498.818380:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050499.713488:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050499.841318:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050500.739634:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050500.865686:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050501.762454:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050501.889523:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050502.017493:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050502.017500:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050502.017503:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050502.017515:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4137 00000100:00080000:1.0:1781050502.017519:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050502.017526:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050502.017536:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050502.017551:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001cbc13b4 x1867562513420608/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050502.017575:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050502.017578:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050502.017607:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513420544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050502.017701:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050502.017707:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513420480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050502.017856:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420608:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050502.024640:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050502.024731:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050502.024743:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420608:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050502.024794:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513420480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050502.024812:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513420544:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050502.785359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050502.913296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050503.809490:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050503.937437:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050504.834211:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050504.962724:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050505.857816:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050505.986165:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050506.881447:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050507.009351:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050507.137602:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050507.137610:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050507.137614:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050507.137647:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4143 00000100:00080000:0.0:1781050507.137651:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050507.137657:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050507.137669:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050507.137699:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513420800/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050507.137723:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050507.137727:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050507.137821:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420800:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050507.138717:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513420672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050507.138789:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050507.138796:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513420736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050507.144423:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050507.144437:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050507.144449:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420800:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050507.144811:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513420736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050507.144823:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513420672:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050507.905554:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050508.033271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050508.932303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050509.057339:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050509.957053:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050510.083324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050510.977459:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050511.105227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050512.001542:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050512.130063:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050512.257490:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050512.257499:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050512.257503:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050512.257515:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4148 00000100:00080000:3.0:1781050512.257518:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781050512.257525:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050512.257557:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050512.257576:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e8445d x1867562513420992/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050512.257602:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050512.257606:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050512.257636:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513420928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050512.257720:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050512.257871:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420992:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050512.258320:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513420864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050512.260613:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513420864:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050512.262181:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513420928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050512.264305:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050512.264316:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050512.264325:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513420992:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050513.026034:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050513.153290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050514.049408:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050514.177445:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050515.074380:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050515.201294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050516.100001:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050516.226069:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050517.125510:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050517.249272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050517.377390:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050517.377398:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050517.377401:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050517.377413:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4153 00000100:00080000:1.0:1781050517.377417:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050517.377423:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050517.377433:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050517.377447:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513421184/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050517.377471:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050517.377474:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050517.377502:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513421120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050517.377502:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421184:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050517.377608:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050517.381933:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513421120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050517.382279:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050517.382366:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050517.382371:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050517.382376:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421184:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050518.145633:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050518.273991:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050519.169304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050519.297229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050520.193302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050520.321390:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050521.218801:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050521.345211:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050522.241756:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050522.369310:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050522.497411:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050522.497419:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050522.497441:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050522.497453:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4158 00000100:00080000:0.0:1781050522.497457:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050522.497463:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050522.497474:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050522.497495:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513421376/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050522.497522:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050522.497526:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050522.497603:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050522.497720:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050522.497727:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050522.498383:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421376:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050522.500349:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050522.500359:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050522.500378:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421376:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050522.502184:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050522.502200:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421248:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:0.0:1781050523.266424:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050523.393256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050524.289486:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050524.417296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050525.313595:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050525.441292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050526.339966:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050526.465229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050527.361559:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050527.489283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050527.617577:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050527.617694:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050527.617701:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050527.617804:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050527.617809:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050527.617813:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050527.617818:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4163 00000100:00080000:1.0:1781050527.617821:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050527.617826:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050527.617837:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050527.617853:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b738af86 x1867562513421568/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050527.617874:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050527.617877:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050527.618750:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421568:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050527.623787:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050527.623808:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050527.624009:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050527.624018:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050527.624025:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421568:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050528.385354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050528.513293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050529.411377:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050529.538290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050530.433994:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050530.561292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050531.459908:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050531.586180:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050532.481666:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050532.609951:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050532.737806:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050532.738738:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050532.738748:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050532.738949:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050532.738955:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050532.738958:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050532.738964:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4168 00000100:00080000:0.0:1781050532.738967:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781050532.738990:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050532.739562:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050532.739598:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513421760/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050532.739624:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050532.739628:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050532.739835:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421760:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050532.740948:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050532.740978:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050532.743654:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050532.743667:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050532.743678:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421760:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050533.506955:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050533.633295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050534.529724:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050534.657407:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050535.553549:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050535.681345:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050536.580918:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050536.705294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050537.602313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050537.729274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050537.858607:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050537.858762:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050537.858768:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050537.858898:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050537.858904:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050537.858908:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050537.858915:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4173 00000100:00080000:1.0:1781050537.858918:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050537.858924:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050537.858935:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050537.858952:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f819707f x1867562513421952/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050537.858972:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050537.858975:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050537.859152:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421952:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050537.861348:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513421888:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050537.861361:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513421824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050537.863820:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050537.863829:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050537.863837:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513421952:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050538.625361:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050538.753263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050539.649382:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050539.777246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050540.674246:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050540.802689:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050541.697813:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050541.825542:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050542.721478:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050542.849298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050542.977581:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050542.977588:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050542.977591:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050542.977601:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4178 00000100:00080000:1.0:1781050542.977618:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1781050542.977625:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513422080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050542.977626:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050542.977640:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050542.977654:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513422144/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050542.977677:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050542.977681:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050542.977716:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513422016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050542.977756:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422144:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050542.984006:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513422080:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050542.984224:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050542.984230:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050542.984251:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422144:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050542.989875:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513422016:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050543.745361:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050543.873250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050544.769361:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050544.898164:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050545.793431:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050545.921243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050546.817967:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050546.946708:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050547.843545:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050547.969234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050548.097386:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050548.097392:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050548.097396:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050548.097406:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4184 00000100:00080000:3.0:1781050548.097409:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781050548.097414:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050548.097422:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050548.097435:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513422336/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050548.097454:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422208:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050548.097458:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050548.097461:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050548.097496:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513422272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050548.097518:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422336:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781050548.100522:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050548.100534:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050548.100544:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422336:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050548.100653:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513422272:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050548.100704:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422208:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050548.866779:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050548.993252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050549.889390:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050550.017253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050550.914162:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050551.041945:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050551.937349:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050552.065290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050552.961320:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050553.089236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050553.218399:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050553.218424:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513422400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050553.218460:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050553.218468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050553.218472:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050553.218479:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4189 00000100:00080000:2.0:1781050553.218483:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781050553.218490:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781050553.218496:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050553.218505:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513422464:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:2.0:1781050553.218558:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050553.218578:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513422528/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050553.218609:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050553.218613:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050553.218845:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422528:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050553.219804:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513422400:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:02020000:1.0:1781050553.221658:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00100000:3.0:1781050553.221921:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513422464:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050553.236463:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050553.236476:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050553.236489:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422528:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050553.985436:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050554.113290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050555.009544:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050555.137253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050556.034206:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050556.161249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050557.057440:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050557.185319:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050558.081759:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050558.209290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050558.337607:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050558.337615:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050558.337618:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050558.337632:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4194 00000100:00080000:3.0:1781050558.337635:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781050558.337641:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050558.337658:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050558.337676:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513422720/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050558.337701:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050558.337705:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050558.337730:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513422656:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050558.337738:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422720:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050558.337834:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050558.345215:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513422656:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050558.346379:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050558.346387:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050558.346410:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422720:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050558.347293:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422592:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050559.109434:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050559.233537:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050560.129282:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050560.257332:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050561.153365:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050561.281328:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050562.177362:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050562.305246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050563.201312:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050563.329293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050563.457423:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422784:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050563.457439:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050563.457445:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050563.457448:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050563.457454:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4199 00000100:00080000:2.0:1781050563.457473:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781050563.457480:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050563.457489:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050563.457505:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050563.457507:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f57815d x1867562513422912/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781050563.457512:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513422848:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050563.457628:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050563.457633:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050563.457745:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422912:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050563.459264:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513422848:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050563.459353:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050563.459358:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050563.459362:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513422912:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050563.460064:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422784:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050564.225375:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050564.355338:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050565.249421:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050565.377293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050566.273700:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050566.401353:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050567.297416:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050567.425351:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050568.321438:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050568.449311:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050568.581473:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050568.581481:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050568.581484:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050568.581497:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4204 00000100:00080000:3.0:1781050568.581500:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1781050568.581514:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050568.581527:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050568.581537:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050568.581551:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003e7c445c x1867562513423104/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050568.581570:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050568.581573:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050568.581606:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050568.581614:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050568.581625:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423104:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050568.584991:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513422976:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050568.585867:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423040:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050568.586058:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050568.586064:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050568.586069:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423104:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:0.0:1781050569.345645:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050569.475231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050570.370580:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050570.497359:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050571.393480:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050571.521281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050572.417467:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050572.545297:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050573.441871:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050573.569563:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050573.697486:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513423168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050573.697888:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050573.697896:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050573.698293:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050573.698465:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050573.698469:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050573.698475:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4209 00000100:00080000:2.0:1781050573.698478:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781050573.698484:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050573.698495:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050573.698531:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4afdb11 x1867562513423296/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781050573.698603:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423296:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050573.698662:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050573.698666:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050573.700176:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513423168:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050573.704108:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050573.704330:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050573.704336:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050573.704341:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423296:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050574.465637:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050574.593251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050575.490022:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050575.617292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050576.513598:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050576.641298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050577.538875:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050577.665292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050578.561389:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050578.689342:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:2.0:1781050578.817451:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513423360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050578.818281:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050578.818290:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050578.818543:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050578.818553:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050578.818556:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050578.818562:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4214 00000100:00080000:3.0:1781050578.818565:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781050578.818572:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050578.818584:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050578.818669:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513423488/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781050578.818727:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423488:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:3.0:1781050578.818783:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050578.818787:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:1.0:1781050578.821409:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050578.821421:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050578.821429:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423488:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050578.823211:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050578.823228:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513423360:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050579.585476:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050579.714005:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050580.609721:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050580.739757:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050581.635628:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050581.761330:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050582.662228:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050582.785484:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050583.681449:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050583.811868:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050583.943623:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050583.943630:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050583.943633:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050583.943645:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4219 00000100:00080000:3.0:1781050583.943649:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781050583.943655:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050583.943667:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050583.943693:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c39def9 x1867562513423680/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050583.943719:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050583.943722:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050583.943770:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423680:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050583.944015:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513423552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050583.944054:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050583.944059:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050583.947095:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513423552:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050583.947585:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050583.947591:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050583.947595:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423680:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050583.952087:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513423616:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050584.705475:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050584.833287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050585.729698:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050585.858742:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050586.753864:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050586.881274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050587.777772:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050587.905585:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050588.801324:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050588.929302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050589.057751:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050589.057759:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050589.057763:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050589.057774:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4224 00000100:00080000:1.0:1781050589.057778:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050589.057784:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050589.057794:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050589.057810:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1c3efb9 x1867562513423872/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050589.057854:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050589.057858:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050589.058364:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423872:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050589.058946:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513423744:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050589.058992:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050589.059009:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513423808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050589.060937:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513423808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050589.060950:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513423744:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050589.063844:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050589.063858:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050589.063866:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513423872:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050589.826661:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050589.953287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050590.849580:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050590.978741:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050591.878644:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050592.002061:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050592.898823:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050593.025964:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050593.922565:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050594.049278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050594.177339:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050594.177347:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050594.177351:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781050594.177355:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513424000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050594.177359:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4230 00000100:00080000:1.0:1781050594.177364:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050594.177371:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050594.177380:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050594.177398:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513424064/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050594.177428:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050594.177432:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050594.177471:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513423936:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050594.178572:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424064:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050594.182835:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050594.182846:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050594.182854:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424064:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050594.182967:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513423936:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050594.184403:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513424000:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050594.946548:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050595.073302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050595.969529:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050596.097276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050596.994769:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050597.122013:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050598.017597:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050598.145279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050599.041586:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050599.169336:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:0.0:1781050599.297489:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513424128:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050599.297756:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050599.297763:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513424192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050599.299414:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050599.299424:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050599.299427:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050599.299436:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4235 00000100:00080000:1.0:1781050599.299439:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781050599.299445:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050599.299457:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050599.299482:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000023b4b231 x1867562513424256/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050599.299511:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050599.299514:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050599.299802:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424256:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050599.307436:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c26c4b8c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513424192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050599.307461:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513424128:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050599.310729:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050599.310742:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050599.310772:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424256:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:0.0:1781050600.065445:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050600.193574:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050601.090541:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050601.217266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050602.117042:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050602.241346:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050603.137386:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050603.265307:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050604.161776:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050604.289399:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050604.417398:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050604.417407:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050604.417410:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781050604.417520:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513424384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050604.417627:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050604.417633:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513424320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050604.417925:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4240 00000100:00080000:1.0:1781050604.417928:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050604.417935:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050604.417947:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050604.417964:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d3f7d01e x1867562513424448/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050604.417993:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050604.417997:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050604.418870:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424448:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050604.420545:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513424320:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050604.423382:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513424384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050604.423382:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050604.423394:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050604.423401:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424448:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050605.186217:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050605.313603:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050606.211351:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050606.337745:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050607.233850:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050607.361276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050608.257424:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050608.385294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050609.281372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050609.409322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050609.537611:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513424512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050609.537785:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050609.537791:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050609.537794:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050609.537799:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4245 00000100:00080000:0.0:1781050609.537802:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050609.537808:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050609.537819:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050609.537833:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513424640/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050609.537882:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050609.537886:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050609.537911:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513424576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050609.537930:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424640:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050609.543618:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513424576:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050609.543698:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513424512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050609.543708:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050609.543715:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050609.543720:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424640:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050610.305479:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050610.433232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050611.329578:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050611.457419:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050612.354066:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050612.481430:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050613.377437:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050613.505302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050614.403215:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050614.529327:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050614.660020:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050614.660027:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050614.660031:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050614.660044:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4250 00000100:00100000:3.0:1781050614.660044:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513424704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050614.660048:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050614.660056:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050614.660084:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050614.660103:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4afdb11 x1867562513424832/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050614.660183:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424832:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050614.660225:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050614.660229:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050614.660257:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050614.660263:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513424768:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050614.665726:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513424704:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050614.665915:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050614.665921:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050614.665926:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513424832:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050614.666903:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513424768:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:0.0:1781050615.425717:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050615.553267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050616.449409:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050616.577424:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050617.473686:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050617.601299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:0.0:1781050618.499246:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050618.625283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050619.521449:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050619.649397:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050619.777938:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050619.777944:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050619.777947:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050619.777956:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4255 00000100:00080000:2.0:1781050619.777960:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050619.777965:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050619.777974:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050619.777985:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c170795f x1867562513425024/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050619.778003:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050619.778006:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050619.778035:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513424960:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050619.778168:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050619.778174:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513424896:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050619.780013:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425024:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781050619.787789:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050619.787800:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050619.787824:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425024:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050619.790470:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513424896:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050619.790515:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513424960:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050620.545391:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050620.673587:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050621.571372:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050621.697494:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050622.594816:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050622.721285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050623.617478:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050623.745769:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050624.644460:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050624.769280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050624.900604:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050624.900612:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050624.900616:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050624.900628:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4260 00000100:00080000:3.0:1781050624.900631:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050624.900638:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050624.900649:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050624.900666:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e944676 x1867562513425216/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050624.900690:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050624.900694:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050624.900723:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513425152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050624.900727:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425216:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050624.900783:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513425088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050624.908371:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050624.908385:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050624.908396:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425216:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050624.908549:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513425088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050624.910459:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513425152:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:3.0:1781050625.667008:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050625.793277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050626.689365:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050626.820189:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050627.713447:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050627.847714:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050628.739514:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050628.865567:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050629.762273:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050629.889240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050630.017380:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050630.017388:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050630.017391:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050630.017402:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4265 00000100:00080000:0.0:1781050630.017405:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050630.017411:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050630.017424:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050630.017440:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dd1a56a6 x1867562513425408/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781050630.017503:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425408:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781050630.017910:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050630.017914:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050630.017939:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513425344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050630.018038:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513425280:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050630.034113:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050630.034165:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050630.034174:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513425344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050630.034174:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425408:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050630.034177:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513425280:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050630.785370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050630.913294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050631.809374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050631.937284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050632.833308:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050632.961316:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050633.857359:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050633.986168:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050634.882936:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050635.009393:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050635.137383:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050635.137389:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050635.137391:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050635.137401:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4271 00000100:00080000:0.0:1781050635.137410:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050635.137417:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050635.137428:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050635.137445:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513425600/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050635.137469:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050635.137473:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050635.137500:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513425536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050635.137589:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050635.137594:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513425472:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050635.137670:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425600:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:3.0:1781050635.145713:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050635.145728:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050635.145739:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425600:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050635.145794:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513425536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050635.145818:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513425472:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050635.905404:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050636.034411:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050636.931269:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050637.057289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050637.953524:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050638.081772:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050638.977371:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050639.105935:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050640.002519:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050640.129334:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050640.257866:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050640.257874:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050640.257877:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050640.257888:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4276 00000100:00080000:0.0:1781050640.257892:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050640.257898:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050640.257910:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050640.257925:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513425792/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050640.257956:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050640.257959:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050640.257989:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513425728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050640.258009:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425792:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050640.258118:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513425664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050640.265603:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050640.265618:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050640.265629:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425792:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050640.268295:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513425664:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050640.268297:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513425728:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:3.0:1781050641.025945:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050641.153327:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050642.052018:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050642.177302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050643.073387:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050643.201481:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050644.097746:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050644.227250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050645.121403:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050645.249782:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050645.383548:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050645.383557:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050645.383560:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050645.383582:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4281 00000100:00080000:2.0:1781050645.383593:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050645.383600:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050645.383610:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050645.383620:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513425856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050645.383718:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050645.383736:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513425920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050645.383802:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513425984/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050645.383864:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425984:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050645.383915:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050645.383919:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050645.387639:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513425920:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050645.390686:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050645.390699:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050645.390709:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513425984:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781050645.395821:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513425856:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050646.146482:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050646.273263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050647.169611:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050647.297267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050648.193424:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050648.321363:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050649.217436:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050649.345974:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050650.241613:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050650.369546:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050650.498338:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050650.498346:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050650.498368:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050650.498381:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4286 00000100:00080000:0.0:1781050650.498384:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050650.498391:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050650.498402:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050650.498421:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002db543f2 x1867562513426176/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050650.498452:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050650.498456:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050650.498486:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513426112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050650.498662:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426176:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050650.498733:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050650.498739:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513426048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050650.505027:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513426048:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050650.506039:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050650.506048:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050650.506055:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426176:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050650.508771:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513426112:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050651.265395:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050651.393285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050652.289699:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050652.417284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050653.314115:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050653.441604:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050654.337412:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050654.466052:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050655.361572:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050655.489371:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050655.617567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050655.617575:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050655.617578:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050655.617590:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4291 00000100:00080000:2.0:1781050655.617593:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050655.617599:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050655.617609:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050655.617624:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513426368/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050655.617653:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050655.617657:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050655.617882:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426368:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050655.618490:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513426240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050655.618529:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050655.618536:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513426304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050655.620938:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513426304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050655.620955:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513426240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050655.621898:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050655.621905:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050655.621912:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426368:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1781050656.385387:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050656.513287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050657.409354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050657.537280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050658.433384:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050658.561307:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050659.460459:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050659.585671:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050660.481447:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050660.620847:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050660.737856:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050660.737864:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050660.737868:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050660.737881:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4296 00000100:00080000:1.0:1781050660.737885:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050660.737891:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050660.737902:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050660.737920:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b12abf0d x1867562513426560/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050660.737947:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050660.737951:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050660.738025:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426560:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050660.738082:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513426496:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050660.738141:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050660.738148:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513426432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050660.741114:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050660.741163:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050660.741172:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426560:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050660.741273:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513426432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050660.741289:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513426496:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050661.505377:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050661.633471:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050662.529565:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050662.657235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050663.556695:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050663.681401:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050664.577462:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050664.707051:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050665.601392:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050665.729291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050665.857406:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050665.857435:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050665.857438:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050665.857451:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4301 00000100:00100000:0.0:1781050665.857452:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513426624:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050665.857456:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050665.857465:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050665.857476:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050665.857494:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513426752/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050665.857524:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050665.857528:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050665.857546:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050665.857555:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513426688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050665.859605:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426752:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050665.863023:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050665.863037:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050665.863048:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426752:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050665.863118:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513426688:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050665.863169:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513426624:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050666.626546:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050666.754356:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050667.649383:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050667.778494:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050668.673750:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050668.803456:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050669.699895:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050669.825268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050670.721395:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050670.849305:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050670.977440:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050670.977448:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050670.977451:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050670.977463:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4306 00000100:00080000:0.0:1781050670.977467:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050670.977475:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050670.977485:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050670.977502:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002db543f2 x1867562513426944/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050670.977530:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050670.977534:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050670.977614:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426944:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050670.977935:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513426816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050670.977983:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050670.977988:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513426880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050670.980404:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513426880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050670.980416:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513426816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050670.980618:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050670.980622:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050670.980625:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513426944:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:3.0:1781050671.745432:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050671.875192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050672.769399:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050672.897266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050673.795174:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050673.921288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050674.820256:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050674.945875:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050675.841443:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050675.969267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050676.097408:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513427008:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050676.097664:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050676.097671:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513427072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050676.098416:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050676.098423:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050676.098426:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050676.098433:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4312 00000100:00080000:2.0:1781050676.098436:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050676.098442:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050676.098453:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050676.098473:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513427136/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781050676.098978:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427136:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050676.099057:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050676.099061:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781050676.103408:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050676.103423:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050676.103434:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427136:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050676.103747:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513427072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050676.103761:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513427008:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050676.865500:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050676.993274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050677.889752:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050678.017259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050678.914290:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050679.042158:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050679.937620:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050680.065285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:3.0:1781050680.961350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050681.090234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050681.218351:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050681.218357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050681.218360:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050681.218369:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4317 00000100:00080000:0.0:1781050681.218372:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050681.218378:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050681.218386:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050681.218397:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513427328/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050681.218416:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050681.218436:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050681.218475:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513427264:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050681.218587:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050681.218590:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427328:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050681.218610:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513427200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050681.221787:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513427200:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050681.221809:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513427264:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050681.221904:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050681.221908:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050681.221912:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427328:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050681.985430:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050682.113837:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050683.011010:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050683.137282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050684.035082:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050684.161295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050685.058977:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050685.185282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050686.081445:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050686.209312:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050686.337373:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1781050686.337378:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513427392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050686.337381:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050686.337385:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050686.337392:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4322 00000100:00080000:0.0:1781050686.337395:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050686.337402:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050686.337411:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050686.337428:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513427520/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050686.337455:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050686.337460:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050686.337490:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050686.337506:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513427456:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050686.337777:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427520:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050686.343638:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513427456:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050686.343662:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513427392:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050686.347447:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050686.347461:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050686.347490:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427520:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050687.106718:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050687.233300:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050688.129505:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050688.257342:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050689.153337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050689.281292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050690.177434:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050690.305282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050691.201397:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050691.330092:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050691.457597:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050691.457606:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050691.457609:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050691.457620:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4327 00000100:00080000:3.0:1781050691.457624:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050691.457630:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050691.457640:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050691.457659:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000062fd0879 x1867562513427712/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050691.457684:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050691.457687:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050691.457717:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513427648:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050691.457814:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050691.457820:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513427584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050691.458312:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427712:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050691.464301:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bb64ba62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513427584:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050691.464328:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f9891000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513427648:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050691.464672:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050691.464677:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050691.464682:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000062fd0879 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427712:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050692.227291:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050692.353298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050693.249396:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050693.377282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050694.275944:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050694.401708:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050695.297345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050695.425184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050696.321401:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050696.449292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050696.577339:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050696.577347:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050696.577350:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050696.577363:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4332 00000100:00080000:3.0:1781050696.577366:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050696.577372:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050696.577387:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050696.577425:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003a9ff07a x1867562513427904/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050696.577441:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050696.577445:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050696.577496:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427904:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050696.578324:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513427776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050696.578375:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050696.578382:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513427840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050696.579773:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050696.579780:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050696.579787:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003a9ff07a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513427904:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050696.580759:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e944676 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513427776:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050696.580781:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c39def9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513427840:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050697.348782:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050697.473248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050698.369370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050698.497253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050699.393365:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050699.521287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050700.418330:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050700.545231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050701.441297:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050701.569273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050701.697402:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050701.697432:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050701.697436:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050701.697448:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4337 00000100:00080000:2.0:1781050701.697451:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050701.697458:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050701.697468:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050701.697485:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000523c4981 x1867562513428096/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050701.697512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050701.697515:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050701.697548:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513428032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050701.697578:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513427968:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050701.697653:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428096:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050701.701227:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513428032:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050701.703699:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050701.703709:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050701.703717:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428096:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050701.703834:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513427968:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050702.467149:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050702.593304:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050703.489594:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050703.617273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050704.516766:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050704.641302:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050705.537425:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050705.667763:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050706.561354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050706.689287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050706.817845:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050706.817853:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050706.817856:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050706.817868:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4342 00000100:00080000:2.0:1781050706.817872:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050706.817878:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050706.817888:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050706.817904:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb943f69 x1867562513428288/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050706.817929:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050706.817952:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050706.818067:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513428160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050706.818204:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050706.818210:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513428224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050706.818591:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428288:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050706.824228:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513428224:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050706.824251:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513428160:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:02020000:0.0:1781050706.824344:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:0.0:1781050706.824357:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050706.824361:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050706.824366:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428288:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050707.585541:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050707.713290:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050708.612808:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050708.737651:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050709.635165:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050709.761285:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050710.658271:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050710.786632:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050711.681336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050711.809264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050711.937465:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050711.937473:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050711.937477:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050711.937487:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4347 00000100:00100000:1.0:1781050711.937489:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513428352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050711.937492:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050711.937500:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050711.937523:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050711.937543:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513428480/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050711.937571:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050711.937575:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050711.937596:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050711.937605:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513428416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050711.937663:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428480:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050711.942418:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513428416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050711.942440:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513428352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050711.942510:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050711.942514:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050711.942518:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428480:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050712.705440:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050712.833358:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050713.729483:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050713.860680:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050714.753414:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050714.881278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050715.777457:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050715.905295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050716.803848:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050716.929314:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050717.057472:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050717.057480:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050717.057483:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050717.057496:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4352 00000100:00080000:3.0:1781050717.057499:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050717.057528:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050717.057539:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050717.057556:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b154b256 x1867562513428672/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050717.057579:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050717.057583:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050717.057602:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513428544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050717.057613:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513428608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050717.057676:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428672:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050717.060469:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513428608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050717.068189:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1781050717.068195:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003e7c445c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513428544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050717.068200:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050717.068208:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b154b256 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428672:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050717.825598:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050717.953358:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050718.849523:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050718.977324:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050719.874329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050720.001315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050720.901168:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050721.025850:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050721.921787:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050722.049467:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050722.177371:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050722.177378:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050722.177382:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050722.177395:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4358 00000100:00080000:3.0:1781050722.177399:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781050722.177404:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513428736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050722.177407:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050722.177420:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050722.177438:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000001dbd998 x1867562513428864/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050722.177463:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050722.177467:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050722.177493:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513428800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050722.177615:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428864:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050722.180996:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050722.181010:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050722.181021:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001dbd998 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513428864:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050722.181610:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513428736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050722.181779:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513428800:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050722.945426:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050723.073742:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050723.973838:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050724.097258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050724.993394:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050725.121260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050726.017938:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050726.146466:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050727.042911:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050727.169298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050727.297432:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050727.297439:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050727.297443:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050727.297454:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4363 00000100:00080000:1.0:1781050727.297458:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050727.297464:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050727.297473:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050727.297487:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000000dbb310 x1867562513429056/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050727.297506:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050727.297508:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050727.297535:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513428992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050727.297716:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513428928:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050727.297945:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429056:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050727.302350:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513428992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050727.302509:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050727.302515:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050727.302520:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429056:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050727.304230:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513428928:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050728.065372:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050728.194106:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050729.089383:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050729.217246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050730.113305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050730.241266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050731.137367:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050731.266236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050732.161551:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050732.289292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050732.417599:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513429120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050732.418754:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050732.418765:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513429184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050732.419228:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050732.419234:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050732.419237:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050732.419244:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4368 00000100:00080000:0.0:1781050732.419247:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050732.419253:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050732.419428:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050732.419448:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513429248/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050732.419467:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050732.419470:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050732.419683:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429248:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050732.420653:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513429120:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050732.420679:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513429184:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050732.421923:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050732.421931:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050732.421938:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429248:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050733.185637:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050733.315198:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050734.211362:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050734.337478:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050735.233371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050735.361375:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050736.257317:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050736.385268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050737.281589:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050737.411200:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050737.537456:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050737.537464:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050737.537467:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050737.537479:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4373 00000100:00080000:2.0:1781050737.537482:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050737.537489:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050737.537499:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050737.537515:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005c3ac388 x1867562513429440/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050737.537574:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429440:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050737.537671:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050737.537675:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050737.537744:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513429312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050737.537784:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050737.537789:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513429376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050737.540039:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb943f69 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513429376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050737.540058:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018a334bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513429312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050737.540392:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050737.540397:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050737.540402:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005c3ac388 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429440:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050738.306303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050738.433459:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050739.329542:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050739.457294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050740.354586:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050740.481293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050741.377825:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050741.505773:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050742.402010:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050742.529435:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050742.657487:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050742.657497:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050742.657500:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050742.657511:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4378 00000100:00080000:0.0:1781050742.657518:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050742.657525:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050742.657551:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050742.657577:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b24224a5 x1867562513429632/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050742.657608:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050742.657614:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050742.657700:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513429568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050742.657791:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050742.657797:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513429504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050742.660247:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429632:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050742.665118:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513429504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050742.665163:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513429568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050742.665814:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050742.665820:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050742.665826:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429632:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050743.428974:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050743.553638:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050744.449639:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050744.578897:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050745.473371:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050745.601711:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050746.497635:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050746.627436:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050747.521515:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050747.649538:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050747.777703:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050747.777711:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1781050747.777837:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513429760:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050747.777932:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1781050747.777937:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513429696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050747.777956:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050747.777964:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4383 00000100:00080000:2.0:1781050747.777967:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050747.777974:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050747.777984:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050747.778003:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f51a738e x1867562513429824/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050747.778051:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429824:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050747.778118:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050747.778122:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:0.0:1781050747.780937:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050747.780949:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050747.780959:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f51a738e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513429824:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050747.782847:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513429696:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050747.782868:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513429760:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050748.545481:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050748.675006:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050749.569737:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050749.697292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050750.593484:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050750.721330:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050751.620924:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050751.745261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050752.642063:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050752.769287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:3.0:1781050752.897356:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513429888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050752.897395:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050752.897399:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050752.897402:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050752.897407:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4388 00000100:00080000:2.0:1781050752.897409:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050752.897414:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050752.897423:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050752.897436:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e5c8472 x1867562513430016/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781050752.897437:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781050752.897443:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513429952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050752.897471:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050752.897474:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050752.897549:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430016:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050752.899237:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513429888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050752.900665:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513429952:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050752.900759:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050752.900765:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050752.900769:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e5c8472 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430016:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050753.667696:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050753.793299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050754.689881:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050754.817484:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050755.715764:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050755.841276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050756.737921:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050756.867476:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050757.761496:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050757.889288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050758.017398:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050758.017407:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050758.017411:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050758.017422:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4393 00000100:00080000:0.0:1781050758.017427:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050758.017448:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781050758.017460:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513430144:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781050758.017462:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050758.017493:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513430208/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050758.017523:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050758.017528:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050758.017576:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050758.017943:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050758.022002:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430080:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050758.025454:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050758.025466:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050758.025476:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430208:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050758.027154:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513430144:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050758.785463:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050758.918730:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050759.809355:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050759.940029:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050760.833630:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050760.962632:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050761.858375:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050761.985313:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050762.882493:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050763.009335:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050763.137464:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050763.137471:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050763.137474:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050763.137484:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4399 00000100:00080000:3.0:1781050763.137487:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050763.137491:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050763.137499:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050763.137557:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c83f9198 x1867562513430400/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050763.137561:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513430272:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050763.137577:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050763.137581:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050763.137601:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513430336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050763.137642:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430400:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050763.140232:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bc445f2a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513430336:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050763.143447:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050763.143461:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050763.143471:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c83f9198 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430400:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050763.149866:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513430272:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050763.905374:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050764.033355:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050764.930234:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050765.057456:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050765.953899:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050766.081291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050766.977497:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050767.105330:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050768.001344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050768.129241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050768.257618:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430464:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050768.258012:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050768.258019:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513430528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050768.258878:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050768.258885:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050768.258903:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050768.258910:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4404 00000100:00080000:0.0:1781050768.258913:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050768.258918:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050768.258929:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050768.258947:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513430592/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050768.258969:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050768.258976:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050768.259000:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430592:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050768.260185:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513430528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050768.260206:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430464:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050768.261060:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050768.261073:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050768.261079:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430592:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050769.025442:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050769.156640:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050770.049518:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050770.179156:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050771.073375:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050771.204516:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050772.097395:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050772.225266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050773.121361:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050773.249289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050773.378073:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050773.378081:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050773.378085:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050773.378099:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4409 00000100:00100000:0.0:1781050773.378100:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430656:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050773.378103:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050773.378112:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781050773.378214:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050773.378222:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513430720:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781050773.378500:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050773.378523:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008bfccf5d x1867562513430784/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050773.378545:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050773.378557:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050773.378610:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050773.382167:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050773.382181:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050773.382191:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430784:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050773.383078:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513430720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050773.383093:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430656:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050774.145527:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050774.273280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050775.169368:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050775.297315:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050776.193290:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050776.321229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050777.217413:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050777.345507:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050778.241371:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050778.371614:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050778.497516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050778.497523:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050778.497526:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050778.497554:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4414 00000100:00080000:1.0:1781050778.497558:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050778.497565:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050778.497575:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050778.497580:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050778.497592:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513430976/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050778.497614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050778.497618:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050778.497640:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513430912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050778.497710:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430976:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1781050778.502265:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050778.502287:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050778.502299:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513430976:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050778.502336:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513430848:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050778.502426:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513430912:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050779.265348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050779.393279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050780.289353:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050780.417270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050781.313422:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050781.441355:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050782.337386:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050782.465650:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050783.363756:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050783.489965:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050783.617360:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050783.617366:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050783.617369:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050783.617379:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4419 00000100:00080000:3.0:1781050783.617383:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050783.617388:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050783.617398:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050783.617399:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513431040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050783.617425:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005840dd0f x1867562513431168/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050783.617470:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050783.617473:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050783.617492:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050783.617501:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513431104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050783.617687:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431168:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050783.619096:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000048c6b8a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513431040:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050783.619165:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b651757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513431104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050783.619332:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050783.619337:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050783.619341:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005840dd0f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431168:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050784.385348:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050784.513264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050785.409251:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050785.537273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050786.433344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050786.561187:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050787.457299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050787.585288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050788.481357:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050788.609277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050788.737356:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050788.737362:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050788.737366:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050788.737378:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4424 00000100:00080000:0.0:1781050788.737382:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050788.737389:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050788.737400:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050788.737406:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050788.737416:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c99a9336 x1867562513431360/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050788.737444:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050788.737447:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050788.737508:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513431296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050788.737516:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050788.738977:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431232:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050788.739019:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050788.739024:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050788.739030:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431360:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050788.739460:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513431296:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050789.505396:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050789.634770:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050790.529836:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050790.658454:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050791.553348:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050791.681238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050792.577344:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050792.705287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050793.601353:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050793.729254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050793.857318:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050793.857327:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050793.857349:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781050793.857350:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431424:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050793.857357:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4429 00000100:00080000:0.0:1781050793.857361:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050793.857368:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050793.857378:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050793.857395:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513431552/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050793.857421:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050793.857425:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050793.857448:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513431488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050793.857915:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431552:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050793.858918:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781050793.858920:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513431488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050793.858924:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050793.858932:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431552:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050793.859176:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431424:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050794.625370:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050794.753282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050795.649379:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050795.777369:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050796.673357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050796.801277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050797.697354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050797.825275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050798.721341:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050798.849275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050798.977399:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050798.977404:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050798.977407:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050798.977417:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4434 00000100:00080000:0.0:1781050798.977420:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050798.977426:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050798.977435:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050798.977443:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050798.977449:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b24224a5 x1867562513431744/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050798.977465:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050798.977468:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050798.977508:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431744:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050798.977525:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513431680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050798.978952:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431616:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050798.978996:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513431680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050798.979026:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050798.979031:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050798.979034:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431744:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050799.745363:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050799.873307:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050800.769525:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050800.897288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050801.793511:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050801.921231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050802.817499:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050802.945279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050803.841719:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050803.969965:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050804.097327:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050804.097336:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050804.097339:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050804.097349:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4440 00000100:00100000:0.0:1781050804.097350:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050804.097352:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050804.097359:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050804.097368:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050804.097383:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a161b523 x1867562513431936/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050804.097418:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050804.097422:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050804.097444:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050804.097452:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513431872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050804.097463:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431936:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050804.098904:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513431872:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050804.098917:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513431808:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050804.098956:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050804.098961:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050804.098966:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513431936:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050804.865311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050804.993235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050805.889340:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050806.017276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050806.913369:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050807.041213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050807.937356:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050808.065292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050808.962164:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050809.089278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050809.217353:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781050809.217384:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513432000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050809.217445:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050809.217449:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050809.217454:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4445 00000100:00080000:1.0:1781050809.217457:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050809.217464:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050809.217474:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050809.217491:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000013bcd30c x1867562513432128/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050809.217509:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1781050809.217511:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050809.217515:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050809.217518:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513432064:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050809.217557:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432128:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050809.222327:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050809.222340:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050809.222349:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432128:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050809.222623:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513432000:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050809.223912:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513432064:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050809.985354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050810.113283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050811.009637:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050811.137288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050812.033569:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050812.161263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050813.057341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050813.185312:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050814.081324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050814.209269:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050814.337751:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050814.337758:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050814.337762:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050814.337774:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4450 00000100:00080000:0.0:1781050814.337777:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050814.337789:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050814.337799:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050814.337834:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513432320/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050814.337854:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050814.337858:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050814.337893:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432320:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050814.339897:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513432256:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050814.340005:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050814.340012:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513432192:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050814.342805:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050814.342830:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050814.342837:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432320:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050814.346723:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513432256:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050814.348174:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513432192:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050815.105700:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050815.233270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050816.129688:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050816.257280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050817.153360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050817.281348:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050818.177614:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050818.305265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050819.201334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050819.329233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050819.457404:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050819.457415:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050819.457419:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050819.457432:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4455 00000100:00080000:3.0:1781050819.457437:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:2.0:1781050819.457440:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050819.457447:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050819.457459:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050819.457479:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b9e1240 x1867562513432512/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050819.457522:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050819.457527:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050819.457556:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513432448:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050819.457606:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432512:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050819.460095:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e8445d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432384:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050819.460099:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050819.460107:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050819.460113:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b9e1240 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432512:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050819.460334:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513432448:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050820.225380:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050820.353261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050821.249298:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050821.377396:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050822.273361:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050822.401242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050823.297264:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050823.425346:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050824.321361:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050824.449283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050824.577363:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050824.577371:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050824.577374:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1781050824.577388:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432576:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050824.577392:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4460 00000100:00080000:3.0:1781050824.577397:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050824.577434:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050824.577446:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050824.577460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000288f7157 x1867562513432704/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050824.577475:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050824.577477:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050824.577492:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513432640:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050824.577511:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432704:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781050824.579163:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000087a1080b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513432640:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050824.579163:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432576:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050824.579181:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050824.579188:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050824.579193:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000288f7157 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432704:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050825.345501:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050825.473276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050826.369323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050826.497217:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050827.393349:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050827.521287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050828.417344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050828.545469:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050829.441454:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050829.569264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050829.697401:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050829.697408:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050829.697412:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050829.697424:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4465 00000100:00080000:3.0:1781050829.697428:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050829.697436:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781050829.697442:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432768:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781050829.697447:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050829.697465:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513432896/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050829.697487:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050829.697490:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050829.697525:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513432832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050829.697530:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432896:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050829.699016:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432768:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050829.699052:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513432832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050829.704374:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050829.704384:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050829.704393:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513432896:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050830.465454:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050830.593282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050831.489286:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050831.617223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050832.513283:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050832.641301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050833.537270:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050833.665234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050834.561315:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050834.689279:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050834.817349:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050834.817358:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050834.817362:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050834.817373:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4470 00000100:00080000:3.0:1781050834.817377:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050834.817386:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:2.0:1781050834.817387:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432960:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:3.0:1781050834.817399:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050834.817417:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000029f3aabc x1867562513433088/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781050834.817449:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050834.817453:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050834.817484:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433088:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050834.817489:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513433024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050834.819488:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050834.819506:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050834.819515:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433088:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781050834.819738:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513432960:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050834.820182:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513433024:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050835.585260:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050835.713286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050836.609329:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050836.739371:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050837.633334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050837.762454:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050838.658535:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050838.786849:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050839.681440:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050839.809277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050839.937351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050839.937360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050839.937364:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050839.937376:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4475 00000100:00080000:0.0:1781050839.937380:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050839.937387:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050839.937398:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050839.937415:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513433280/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050839.937459:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050839.937463:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050839.937747:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433280:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050839.939772:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050839.939781:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050839.939789:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433280:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050839.940209:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433152:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050839.940280:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050839.940288:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513433216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050839.941755:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513433216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050839.942042:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433152:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050840.705303:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050840.833296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050841.729317:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050841.857253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050842.753327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050842.881250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050843.777333:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050843.905319:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050844.801333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1781050844.930842:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050845.057386:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781050845.057416:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050845.057416:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050845.057421:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050845.057429:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4480 00000100:00080000:1.0:1781050845.057432:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050845.057439:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050845.057450:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050845.057468:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b2b72c7a x1867562513433472/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050845.057493:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050845.057496:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050845.057533:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513433408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050845.057546:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433472:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050845.062775:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e41ff85d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433344:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050845.062803:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513433408:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050845.067188:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050845.067199:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050845.067206:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b2b72c7a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433472:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050845.825350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050845.953354:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050846.849344:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050846.977287:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050847.873350:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050848.001292:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050848.897318:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050849.025293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050849.921342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050850.049251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050850.177889:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050850.177897:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050850.177900:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050850.177912:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4486 00000100:00100000:1.0:1781050850.177914:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513433600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050850.177916:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050850.177926:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050850.177937:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050850.177954:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513433664/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050850.177998:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050850.178020:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050850.178048:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050850.178056:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513433536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050850.179071:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433664:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050850.179607:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513433536:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050850.179625:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513433600:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050850.180244:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050850.180259:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050850.180264:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433664:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050850.945347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050851.073289:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050851.969345:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050852.097227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050852.993472:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050853.121706:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050854.017390:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050854.146060:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050855.041357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050855.169255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050855.297467:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050855.297473:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050855.297475:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050855.297487:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4491 00000100:00080000:1.0:1781050855.297491:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050855.297498:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050855.297508:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050855.297533:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a161b523 x1867562513433856/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050855.297557:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050855.297561:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050855.297594:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513433792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050855.297616:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433856:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050855.297677:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050855.300415:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003ce8e0ca pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433728:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050855.300439:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513433792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050855.300631:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050855.300636:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050855.300641:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a161b523 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513433856:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050856.065342:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050856.193265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050857.089318:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050857.217273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050858.113278:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050858.243038:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050859.137428:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050859.266085:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050860.161417:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050860.289277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050860.417637:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050860.417647:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050860.417651:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050860.417663:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4496 00000100:00080000:0.0:1781050860.417666:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050860.417676:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781050860.417676:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433920:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781050860.417695:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050860.417721:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513434048/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050860.417743:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050860.417746:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050860.417779:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050860.417804:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434048:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050860.417821:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513433984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:2.0:1781050860.419334:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:2.0:1781050860.419346:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050860.419350:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050860.419356:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434048:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050860.419357:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513433984:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050860.419562:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513433920:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050861.186423:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050861.313291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050862.209360:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050862.337242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050863.233472:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050863.361268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050864.257308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050864.388914:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050865.281311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050865.409240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050865.537306:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050865.537319:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050865.537323:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050865.537335:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4501 00000100:00080000:0.0:1781050865.537339:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781050865.537342:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513434112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050865.537348:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050865.537363:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050865.537390:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dd1a56a6 x1867562513434240/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050865.537436:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050865.537439:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050865.537460:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513434176:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050865.537568:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434240:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050865.542411:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050865.542423:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050865.542436:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434240:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050865.542514:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513434112:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050865.543805:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513434176:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050866.305440:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050866.433513:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050867.333854:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050867.457278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050868.353383:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050868.481266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050869.381001:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050869.505286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050870.402719:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050870.529258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050870.657540:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050870.657548:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050870.657551:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050870.657563:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4506 00000100:00080000:0.0:1781050870.657566:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050870.657573:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050870.657583:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050870.657597:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513434432/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050870.657633:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050870.657637:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050870.657708:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434432:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050870.657753:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513434304:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050870.658079:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513434368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050870.662086:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513434368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050870.665050:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050870.665063:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050870.665073:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434432:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050870.669513:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513434304:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050871.427183:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050871.557171:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050872.449413:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050872.577288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050873.473324:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050873.603186:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050874.498568:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050874.626110:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050875.521608:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050875.649616:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050875.780624:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050875.780632:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050875.780636:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050875.780649:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4511 00000100:00080000:2.0:1781050875.780653:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050875.780660:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050875.780672:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050875.780688:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007da72ee4 x1867562513434624/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1781050875.780842:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434624:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050875.780953:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050875.780957:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050875.780981:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513434560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050875.781020:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050875.781026:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513434496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050875.782625:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050875.782635:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050875.782651:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513434496:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050875.782660:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007da72ee4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434624:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050875.783240:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513434560:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050876.545352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050876.673401:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050877.572588:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050877.697281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050878.593420:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050878.721991:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050879.617724:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050879.745835:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050880.645508:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050880.769251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050880.897306:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050880.897315:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050880.897319:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050880.897331:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4516 00000100:00080000:2.0:1781050880.897335:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:3.0:1781050880.897340:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513434688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050880.897344:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050880.897359:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050880.897388:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398b1a09 x1867562513434816/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050880.897413:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050880.897417:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050880.897466:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513434752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050880.897482:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434816:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050880.900199:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513434688:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050880.904666:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050880.904677:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050880.904688:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398b1a09 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513434816:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050880.906463:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513434752:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050881.665988:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050881.793281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050882.689436:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050882.817286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050883.713435:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050883.842164:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050884.738898:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050884.865423:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050885.761362:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050885.889264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050886.017469:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513434880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050886.017632:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050886.017637:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513434944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050886.017724:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050886.017734:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050886.017738:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050886.017744:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4521 00000100:00080000:0.0:1781050886.017748:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050886.017754:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050886.017767:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050886.017787:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513435008/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050886.017818:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050886.017822:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050886.017890:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435008:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050886.019458:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513434944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050886.019599:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513434880:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050886.019730:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050886.019735:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050886.019741:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435008:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050886.785346:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050886.913270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050887.809457:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050887.937446:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050888.836594:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050888.961253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050889.857313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050889.985283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050890.881383:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050891.009257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050891.138507:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050891.138515:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050891.138519:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050891.138531:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4527 00000100:00080000:0.0:1781050891.138534:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781050891.138538:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513435072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050891.138542:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050891.138572:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050891.138592:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009fa84b8b x1867562513435200/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050891.138612:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050891.138615:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050891.138630:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513435136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050891.138642:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435200:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050891.143687:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513435072:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050891.143864:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050891.143869:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050891.143873:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435200:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050891.143960:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513435136:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050891.905549:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050892.033291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050892.929355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050893.057322:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050893.953347:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050894.081293:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050894.977362:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050895.105580:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050896.001314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050896.129273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050896.257439:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050896.257445:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050896.257448:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050896.257456:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4532 00000100:00080000:0.0:1781050896.257458:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050896.257463:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050896.257470:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050896.257481:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002db543f2 x1867562513435392/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050896.257500:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050896.257503:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050896.257540:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513435328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050896.257614:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050896.257618:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513435264:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050896.257718:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050896.263933:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050896.263945:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050896.263954:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435392:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050896.264166:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513435328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050896.264185:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513435264:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050897.025314:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050897.153256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050898.049346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050898.177254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050899.073290:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050899.201250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050900.098422:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050900.225342:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050901.121373:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050901.249262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050901.378042:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050901.378049:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050901.378052:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050901.378097:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4537 00000100:00080000:2.0:1781050901.378100:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050901.378106:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050901.378116:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050901.378164:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000e018bdc x1867562513435584/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050901.378179:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050901.378182:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050901.378202:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513435520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050901.378342:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050901.378349:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513435456:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050901.378784:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435584:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781050901.382018:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050901.382029:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050901.382039:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000e018bdc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435584:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050901.382377:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513435520:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050901.382400:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513435456:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050902.145436:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050902.273235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050903.169301:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050903.297249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050904.193299:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050904.321463:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050905.217583:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050905.345239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050906.242028:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050906.369220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050906.497377:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050906.497384:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050906.497387:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050906.497396:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4542 00000100:00080000:1.0:1781050906.497398:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781050906.497401:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513435648:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050906.497440:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050906.497465:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050906.497478:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ea0bf1d5 x1867562513435776/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050906.497504:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050906.497508:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050906.497509:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050906.497518:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513435712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050906.498026:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1781050906.500688:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050906.500696:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050906.500703:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435776:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050906.501879:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000129aa103 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513435712:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050906.501892:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000013bcd30c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513435648:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050907.265288:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050907.393247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050908.289366:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050908.417601:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050909.317498:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050909.448175:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050910.337957:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050910.465295:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050911.363967:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050911.489433:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050911.617407:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513435840:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050911.617572:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050911.617579:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513435904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050911.617713:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050911.617719:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050911.617722:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050911.617727:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4547 00000100:00080000:0.0:1781050911.617730:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050911.617735:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050911.617744:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050911.617758:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513435968/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1781050911.617829:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435968:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050911.617924:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050911.617929:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00080000:2.0:1781050911.625891:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050911.625903:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050911.625915:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513435904:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050911.625915:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513435968:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050911.625935:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513435840:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050912.385367:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050912.513284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050913.409334:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050913.537191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050914.433256:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050914.561271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050915.457351:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050915.585448:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050916.482800:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050916.609260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050916.737408:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050916.737427:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050916.737430:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050916.737441:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4552 00000100:00080000:1.0:1781050916.737445:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050916.737451:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050916.737462:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050916.737477:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c3832bdb x1867562513436160/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050916.737504:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050916.737507:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050916.737535:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513436096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050916.737630:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1781050916.737637:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513436032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050916.737654:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436160:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781050916.743561:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050916.743572:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050916.743595:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c3832bdb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436160:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781050916.743693:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513436032:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781050916.743708:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513436096:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050917.505341:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050917.633266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050918.529303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050918.657268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050919.553315:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050919.681245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050920.577358:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050920.705236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050921.601337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050921.729253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050921.857350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1781050921.857392:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050921.857481:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1781050921.857486:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513436288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050921.858528:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050921.858531:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050921.858536:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4557 00000100:00080000:1.0:1781050921.858539:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050921.858544:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050921.858553:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050921.858572:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009b3dd4d8 x1867562513436352/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050921.858596:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050921.858599:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050921.858644:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436352:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050921.859396:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d7e018b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436224:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050921.859418:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ea0bf1d5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513436288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050921.859679:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050921.859683:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050921.859687:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009b3dd4d8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436352:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050922.625337:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050922.753261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050923.649389:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050923.777256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050924.673313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050924.803151:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050925.697454:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050925.825252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050926.723837:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050926.849233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781050926.979391:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1781050926.979443:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513436416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050926.979543:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781050926.979550:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513436480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781050926.979800:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781050926.979804:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781050926.979810:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4562 00000100:00080000:3.0:1781050926.979812:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:3.0:1781050926.979818:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781050926.979828:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781050926.979846:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513436544/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050926.979891:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436544:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:3.0:1781050926.979929:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781050926.979934:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781050926.985580:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513436480:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050926.985600:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513436416:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050926.985671:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050926.985676:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050926.985681:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436544:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050927.745348:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050927.873430:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050928.769319:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050928.898847:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050929.793313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050929.921228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050930.817336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050930.945377:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050931.841320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050931.969465:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050932.097317:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050932.097326:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050932.097329:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050932.097340:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4568 00000100:00080000:0.0:1781050932.097344:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781050932.097348:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050932.097352:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050932.097366:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050932.097383:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000728de7f4 x1867562513436736/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050932.097447:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436736:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050932.097491:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050932.097507:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513436672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050932.097554:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050932.097559:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050932.099422:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513436672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050932.100497:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436608:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050932.100585:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050932.100591:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050932.100596:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436736:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050932.865376:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050932.993259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050933.889402:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050934.017270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050934.913672:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050935.041250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050935.937390:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050936.065298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050936.961365:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050937.089591:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050937.217447:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050937.217458:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050937.217485:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781050937.217490:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513436864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050937.217496:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4573 00000100:00080000:0.0:1781050937.217500:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050937.217507:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050937.217521:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050937.217540:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dd1a56a6 x1867562513436928/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050937.217577:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050937.217581:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050937.217614:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050937.218490:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436928:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050937.220248:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513436864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050937.220285:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436800:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050937.220488:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050937.220494:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050937.220499:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513436928:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050937.985466:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050938.113283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050939.009386:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050939.137672:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050940.033383:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050940.163463:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050941.057341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050941.185274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050942.083369:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050942.209231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050942.337712:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050942.337723:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050942.337727:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050942.337740:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4578 00000100:00080000:1.0:1781050942.337743:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050942.337750:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050942.337763:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050942.337780:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a5a8ed39 x1867562513437120/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050942.337826:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050942.337839:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050942.337871:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050942.337929:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050942.337932:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437120:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050942.342473:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008bfccf5d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437056:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050942.342526:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001cbc13b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513436992:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:1.0:1781050942.342561:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050942.342569:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050942.342575:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437120:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050943.105419:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050943.233270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050944.129380:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050944.257256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050945.155176:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050945.281466:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050946.177444:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050946.305243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050947.201407:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1781050947.329280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050947.457298:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050947.457303:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050947.457306:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050947.457314:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4583 00000100:00080000:0.0:1781050947.457316:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050947.457321:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050947.457336:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050947.457370:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513437312/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050947.457392:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050947.457395:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050947.457422:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050947.457439:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437312:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050947.457525:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050947.459306:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050947.459319:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437184:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050947.460207:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781050947.460212:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781050947.460216:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437312:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050948.225370:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050948.355557:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050949.250034:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050949.377276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050950.273336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050950.401283:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050951.297271:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050951.425265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050952.321354:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050952.449357:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050952.577344:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050952.577353:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050952.577357:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050952.577371:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4588 00000100:00100000:1.0:1781050952.577371:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050952.577376:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050952.577385:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050952.577396:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050952.577416:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513437504/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050952.577442:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050952.577445:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050952.577466:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050952.577493:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437504:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050952.578846:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437376:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050952.578856:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1781050952.578857:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050952.578862:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050952.578866:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437504:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050953.348255:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050953.473267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050954.369287:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050954.497294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050955.393313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050955.522890:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050956.417289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050956.545311:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050957.441593:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050957.569240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050957.697361:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050957.697368:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050957.697387:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050957.697400:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4593 00000100:00100000:1.0:1781050957.697401:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050957.697405:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050957.697414:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050957.697429:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050957.697445:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513437696/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050957.697464:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050957.697466:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050957.697480:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050957.697522:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437696:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050957.699509:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437568:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050957.699658:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781050957.699714:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781050957.699719:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781050957.699724:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437696:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050958.465364:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050958.593258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050959.489395:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050959.617591:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050960.513394:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050960.641251:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050961.537409:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050961.665245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050962.562313:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050962.689260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050962.817832:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050962.817839:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050962.817842:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050962.817853:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4598 00000100:00080000:0.0:1781050962.817856:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050962.817862:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050962.817872:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050962.817887:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004437141e x1867562513437888/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050962.817914:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050962.817917:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050962.817956:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050962.818069:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050962.818076:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513437760:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781050962.820184:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437888:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050962.822179:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513437760:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781050962.822203:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513437824:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050962.822303:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050962.822321:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050962.822326:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513437888:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050963.585422:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050963.713221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050964.610002:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050964.737277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050965.633450:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050965.761246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050966.657334:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050966.785253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050967.682149:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050967.809260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050967.937367:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050967.937380:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050967.937383:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050967.937394:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4603 00000100:00080000:0.0:1781050967.937398:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050967.937404:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050967.937416:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050967.937431:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513438080/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050967.937456:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050967.937460:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050967.937502:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513438016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050967.937517:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438080:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781050967.937638:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437952:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050967.939538:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050967.939547:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050967.939555:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438080:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050967.939581:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513438016:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050967.939832:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513437952:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050968.706101:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050968.833257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050969.729340:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050969.857262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050970.753338:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050970.881253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050971.781993:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050971.905264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050972.801375:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050972.929296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050973.057363:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050973.057373:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050973.057378:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050973.057392:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4608 00000100:00100000:3.0:1781050973.057393:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513438144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050973.057397:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050973.057406:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050973.057420:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050973.057439:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d2849ba3 x1867562513438272/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050973.057463:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050973.057466:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050973.057484:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513438208:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050973.057510:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438272:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050973.058822:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513438208:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781050973.058836:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084e7220c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513438144:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050973.058864:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050973.058887:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050973.058890:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d2849ba3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438272:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050973.825343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050973.953246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050974.849540:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050974.977481:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050975.873311:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050976.002859:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050976.897313:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050977.025272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050977.922010:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050978.049286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781050978.177352:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050978.177375:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781050978.177378:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781050978.177390:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4614 00000100:00080000:2.0:1781050978.177393:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781050978.177407:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781050978.177419:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781050978.177434:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000015c9d1f3 x1867562513438464/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781050978.177452:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781050978.177455:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781050978.177492:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513438400:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050978.177608:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781050978.177614:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513438336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050978.178218:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438464:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781050978.179353:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513438336:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050978.180505:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513438400:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781050978.181997:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781050978.182008:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781050978.182015:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000015c9d1f3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438464:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781050978.945367:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050979.073282:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050979.969730:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050980.097394:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050980.993484:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050981.121278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050982.017489:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050982.145291:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050983.041343:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050983.169266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781050983.297440:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050983.297451:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050983.297457:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050983.297472:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4619 00000100:00080000:0.0:1781050983.297476:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050983.297484:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050983.297499:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050983.297522:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513438656/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781050983.297563:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050983.297567:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781050983.297618:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513438592:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781050983.297761:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781050983.297770:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513438528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781050983.297847:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438656:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050983.301428:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513438528:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781050983.302117:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050983.302164:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050983.302171:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438656:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781050983.302686:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513438592:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781050984.065820:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050984.193294:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050985.089336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050985.217261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050986.113857:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050986.241306:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050987.137435:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050987.265265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050988.161319:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050988.289253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050988.417332:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050988.417341:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050988.417345:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050988.417355:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4624 00000100:00100000:0.0:1781050988.417356:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513438784:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050988.417360:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:1.0:1781050988.417368:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050988.417377:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050988.417395:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b98b0b8e x1867562513438848/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050988.417438:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050988.417442:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050988.417474:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513438720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050988.417577:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438848:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050988.419733:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781050988.419734:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513438720:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050988.419742:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050988.419747:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b98b0b8e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513438848:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050988.419846:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000000dbb310 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513438784:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050989.185334:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050989.313268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050990.209333:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050990.337257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050991.233350:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050991.361255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050992.257687:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050992.385261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050993.281308:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050993.409467:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00100000:1.0:1781050993.537507:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513438912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050993.537990:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781050993.537996:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513438976:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781050993.538468:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050993.538472:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781050993.538475:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781050993.538481:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4629 00000100:00080000:0.0:1781050993.538483:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781050993.538489:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781050993.538498:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781050993.538515:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b24224a5 x1867562513439040/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1781050993.538553:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439040:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1781050993.538599:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781050993.538602:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050993.539937:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513438912:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050993.541363:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513438976:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781050993.543680:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050993.543693:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050993.543703:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439040:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781050994.305503:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050994.433257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050995.329337:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050995.457280:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050996.354940:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050996.481227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050997.378337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050997.505252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781050998.401419:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050998.529253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781050998.657469:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050998.657476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781050998.657480:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781050998.657491:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4634 00000100:00080000:1.0:1781050998.657494:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:0.0:1781050998.657501:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513439104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781050998.657502:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781050998.657515:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781050998.657534:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef29e3c6 x1867562513439232/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781050998.657554:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781050998.657557:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781050998.657575:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513439168:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781050998.657595:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439232:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1781050998.660778:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781050998.660788:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781050998.660799:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef29e3c6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439232:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781050998.660872:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a5a8ed39 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513439104:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781050998.660952:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513439168:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781050999.425545:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781050999.553261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051000.449469:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051000.577299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051001.473315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051001.601237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051002.497342:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051002.625242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051003.521327:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051003.649573:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051003.778974:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051003.778981:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051003.778984:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051003.778995:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4639 00000100:00080000:2.0:1781051003.778999:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781051003.779006:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051003.779016:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051003.779031:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513439424/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051003.779059:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051003.779078:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781051003.779106:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513439360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051003.779233:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781051003.779240:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513439296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051003.779302:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439424:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781051003.785278:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513439296:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051003.785303:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513439360:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051003.792927:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051003.792937:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051003.792946:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439424:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051004.545372:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051004.673299:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051005.569341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051005.697261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051006.593354:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051006.721261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051007.617323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051007.745261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051008.641302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051008.769263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051008.897333:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051008.897340:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051008.897343:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051008.897354:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4644 00000100:00080000:0.0:1781051008.897357:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781051008.897363:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781051008.897374:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513439552:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781051008.897381:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051008.897396:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513439616/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051008.897420:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051008.897423:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051008.897456:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513439488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051008.897563:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439616:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781051008.901342:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513439488:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781051008.901356:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051008.901374:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051008.901379:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439616:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781051008.902525:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513439552:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051009.665313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051009.793265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051010.689312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051010.817246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051011.713325:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051011.841236:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051012.737332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051012.865228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051013.761322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051013.889209:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051014.017368:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051014.017377:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051014.017382:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781051014.017389:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513439680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051014.017391:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4649 00000100:00080000:0.0:1781051014.017395:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781051014.017404:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051014.017415:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051014.017433:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000caff173d x1867562513439808/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051014.017476:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051014.017480:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051014.017507:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513439744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051014.017521:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439808:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781051014.018894:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c99a9336 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513439680:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051014.018940:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513439744:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:02020000:2.0:1781051014.018941:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:2.0:1781051014.018950:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781051014.018954:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781051014.018958:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000caff173d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513439808:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051014.787109:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051014.913288:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051015.809324:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051015.937259:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051016.833364:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051016.961267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051017.857326:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051017.985235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051018.881314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051019.009246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051019.137339:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051019.137346:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051019.137350:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051019.137360:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4655 00000100:00100000:3.0:1781051019.137363:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513439872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781051019.137364:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781051019.137373:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051019.137383:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051019.137399:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000521535b2 x1867562513440000/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051019.137422:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051019.137440:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781051019.137450:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781051019.137453:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440000:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781051019.137459:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513439936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051019.138653:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513439936:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051019.138670:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513439872:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051019.138742:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051019.138747:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051019.138752:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440000:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051019.905315:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051020.033232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051020.929335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051021.057265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051021.953366:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051022.081267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051022.977443:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051023.105281:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051024.001661:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051024.129286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051024.257409:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051024.257418:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051024.257422:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051024.257434:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4660 00000100:00080000:0.0:1781051024.257438:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00100000:1.0:1781051024.257444:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051024.257446:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051024.257461:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051024.257481:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004708ca3d x1867562513440192/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051024.257506:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051024.257510:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051024.257537:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440128:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051024.257547:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440192:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:2.0:1781051024.261789:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781051024.261795:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003cae67cc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440064:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781051024.261803:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781051024.261811:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004708ca3d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440192:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781051024.261832:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440128:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051025.025401:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051025.153286:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051026.049361:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051026.177254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051027.073358:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051027.201270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051028.097345:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051028.225942:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051029.121385:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051029.249262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051029.377347:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1781051029.377354:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051029.377358:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051029.377362:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051029.377368:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4665 00000100:00080000:0.0:1781051029.377371:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781051029.377377:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051029.377388:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051029.377406:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3949757 x1867562513440384/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051029.377433:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051029.377437:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051029.377477:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051029.377540:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440384:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781051029.382365:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440320:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781051029.382373:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b24224a5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440256:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781051029.382610:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781051029.382617:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781051029.382622:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3949757 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440384:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051030.145357:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051030.274535:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051031.171729:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051031.297301:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051032.193322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051032.321272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051033.217350:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051033.345243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051034.241511:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051034.369238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051034.497367:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051034.497374:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051034.497378:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051034.497390:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4670 00000100:00080000:0.0:1781051034.497393:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781051034.497398:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051034.497408:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051034.497438:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f80d1d9 x1867562513440576/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051034.497456:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051034.497459:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051034.497483:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440512:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051034.497500:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440576:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781051034.497591:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781051034.500514:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781051034.500530:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781051034.500539:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d26e1cb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440448:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051034.500539:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f80d1d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440576:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781051034.500574:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440512:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051035.265266:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051035.393252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051036.289431:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051036.417298:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051037.313310:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051037.441239:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051038.337329:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051038.465223:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051039.361354:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051039.489249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051039.617319:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051039.617326:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051039.617330:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051039.617341:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4675 00000100:00080000:0.0:1781051039.617345:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781051039.617354:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781051039.617355:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440640:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1781051039.617364:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051039.617383:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003c180bf3 x1867562513440768/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051039.617405:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051039.617409:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051039.617431:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440704:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781051039.617456:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440768:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051039.619006:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440640:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051039.619007:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513440704:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781051039.619100:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781051039.619105:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781051039.619109:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440768:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051040.385900:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051040.513243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051041.409312:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051041.537305:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051042.433341:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051042.561246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051043.457282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051043.585268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051044.481485:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051044.609247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051044.738177:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051044.738185:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1781051044.738204:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051044.738208:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051044.738216:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4680 00000100:00080000:0.0:1781051044.738219:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:0.0:1781051044.738226:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051044.738237:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051044.738255:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513440960/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051044.738282:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051044.738286:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781051044.738308:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781051044.738315:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513440896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781051044.738609:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440960:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051044.739863:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513440896:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781051044.739878:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513440832:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781051044.740258:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1781051044.740263:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:2.0:1781051044.740268:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513440960:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051045.505289:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051045.633254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051046.529332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051046.657195:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051047.553345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051047.681275:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051048.577391:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051048.705385:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051049.601292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051049.729261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051049.857556:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051049.857564:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051049.857568:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051049.857579:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4685 00000100:00080000:2.0:1781051049.857584:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 21s 00000100:00080000:2.0:1781051049.857591:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051049.857602:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051049.857621:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513441152/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1781051049.857629:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513441024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781051049.857643:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051049.857653:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051049.857687:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441152:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1781051049.857687:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513441088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781051049.859399:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513441088:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051049.859421:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513441024:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051049.859458:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051049.859466:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051049.859472:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441152:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051050.625322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051050.753258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051051.649440:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051051.777260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051052.673410:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051052.801265:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051053.697385:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051053.825230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051054.721392:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051054.849257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051054.977337:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051054.977374:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051054.977378:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051054.977388:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4690 00000100:00080000:2.0:1781051054.977410:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 6s 00000100:00080000:2.0:1781051054.977416:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051054.977426:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051054.977441:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c170795f x1867562513441344/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051054.977467:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051054.977471:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781051054.977497:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513441280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781051054.977588:0:2216:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1781051054.977594:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513441216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051054.978055:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441344:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781051054.979526:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:a331c586-428d-4036-9702-69d922f5fec1:2216:1867562513441216:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051054.979543:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513441280:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051054.979577:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051054.979581:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051054.979586:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441344:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051055.745321:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051055.873231:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051056.769270:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051056.897248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051057.793335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051057.921208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051058.817313:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051058.945238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051059.841345:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051059.969220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051060.097344:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051060.097351:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051060.097355:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051060.097364:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4696 00000100:00080000:0.0:1781051060.097368:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 11s 00000100:00100000:1.0:1781051060.097374:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513441408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051060.097389:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051060.097399:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051060.097412:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000398ab629 x1867562513441536/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051060.097433:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051060.097436:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051060.097457:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513441472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:2.0:1781051060.097466:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441536:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051060.098771:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513441408:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051060.098791:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513441472:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051060.098843:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051060.098855:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051060.098859:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441536:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051060.865309:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051060.993258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051061.889296:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051062.017255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051062.913327:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051063.041235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051063.937317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051064.065262:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051064.962237:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051065.089266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051065.217298:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051065.217305:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051065.217308:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051065.217318:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4701 00000100:00080000:2.0:1781051065.217322:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 16s 00000100:00100000:3.0:1781051065.217325:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513441600:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:2.0:1781051065.217330:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051065.217340:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051065.217353:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e12224a x1867562513441728/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051065.217370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051065.217373:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051065.217395:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441728:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781051065.217401:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781051065.217410:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513441664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051065.218646:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513441664:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051065.218647:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051065.218651:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051065.218656:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441728:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1781051065.218660:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513441600:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781051065.985333:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051066.113191:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051067.009291:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051067.137221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051068.033297:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051068.161249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051069.057335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051069.185247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051070.081317:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051070.209254:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051070.337309:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051070.337316:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051070.337318:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051070.337327:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4706 00000100:00080000:2.0:1781051070.337331:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781051070.337336:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781051070.337349:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513441792:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781051070.337350:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051070.337362:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009e12224a x1867562513441920/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051070.337377:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051070.337380:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781051070.337397:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513441856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051070.337432:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441920:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781051070.338749:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513441856:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051070.338884:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c91281b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513441792:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051070.338936:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051070.338939:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051070.338942:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009e12224a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513441920:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051071.105307:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051071.233258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051072.129309:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051072.257255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051073.153323:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051073.281221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051074.177247:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051074.305230:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051075.201323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051075.329227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781051075.457322:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781051075.457328:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781051075.457332:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781051075.457342:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4711 00000100:00080000:1.0:1781051075.457345:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781051075.457352:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:0.0:1781051075.457354:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442048:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:1.0:1781051075.457362:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781051075.457380:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ae737f2 x1867562513442112/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781051075.457417:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781051075.457420:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051075.457450:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781051075.457459:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513441984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051075.457463:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442112:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051075.458875:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b738af86 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513441984:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051075.458890:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051075.458894:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051075.458898:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ae737f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442112:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051075.458907:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f819707f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442048:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781051076.225306:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051076.353213:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051077.249295:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051077.378196:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051078.273310:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051078.401243:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051079.297296:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051079.425246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051080.321380:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051080.449256:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051080.577292:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051080.577307:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051080.577310:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781051080.577314:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513442176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051080.577322:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4716 00000100:00080000:0.0:1781051080.577327:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051080.577333:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051080.577342:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051080.577359:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000059ddb857 x1867562513442304/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051080.577377:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051080.577380:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051080.577400:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051080.577426:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442304:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051080.578559:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513442176:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051080.578780:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442240:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051080.578806:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051080.578811:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051080.578816:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442304:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051081.345311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051081.473238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051082.370689:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051082.497240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051083.393355:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051083.521261:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051084.417376:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051084.545328:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051085.441335:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051085.569879:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781051085.697333:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781051085.697341:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781051085.697344:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1781051085.697363:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781051085.697369:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4721 00000100:00080000:1.0:1781051085.697373:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1781051085.697379:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781051085.697389:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781051085.697406:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000084471047 x1867562513442496/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781051085.697434:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781051085.697448:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051085.697458:0:2215:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1781051085.697467:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513442368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051085.697633:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442496:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051085.699117:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b12abf0d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:a331c586-428d-4036-9702-69d922f5fec1:2215:1867562513442368:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051085.699158:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d3f7d01e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442432:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051085.699199:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051085.699212:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051085.699216:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000084471047 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442496:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051086.465504:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051086.593276:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051087.489275:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051087.617855:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051088.513320:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051088.641238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051089.537324:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051089.665228:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051090.561268:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051090.689220:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051090.817306:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051090.817313:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051090.817316:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051090.817325:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4726 00000100:00080000:2.0:1781051090.817328:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781051090.817335:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:3.0:1781051090.817336:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513442560:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:2.0:1781051090.817344:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051090.817356:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f57815d x1867562513442688/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051090.817400:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051090.817404:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781051090.817425:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513442624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051090.817441:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442688:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781051090.818898:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513442624:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051090.818898:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4afdb11 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513442560:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051090.818910:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051090.818920:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051090.818926:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f57815d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442688:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051091.585310:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051091.713238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051092.609314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051092.737219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051093.633285:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051093.761233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051094.657324:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051094.785235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051095.681429:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051095.809268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051095.938959:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051095.938968:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051095.938972:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051095.938985:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4731 00000100:00100000:1.0:1781051095.938986:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513442752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051095.938990:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051095.938998:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051095.939009:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051095.939032:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000018617087 x1867562513442880/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051095.939056:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051095.939060:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051095.939085:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051095.939118:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442880:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781051095.940623:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513442752:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051095.940700:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513442816:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781051095.940701:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051095.940713:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051095.940724:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513442880:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051096.705336:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051096.833334:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051097.729385:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051097.857260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051098.753626:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051098.881266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051099.777360:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051099.905278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051100.801395:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051100.929399:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051101.057349:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051101.057358:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051101.057361:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051101.057373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4736 00000100:00080000:0.0:1781051101.057377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051101.057398:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781051101.057409:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513442944:192.168.201.144@tcp:400:kworker/u8:0.0 00000008:00100000:0.0:1781051101.057410:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051101.057429:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513443072/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051101.057451:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051101.057463:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051101.057512:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443008:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051101.058162:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:1.0:1781051101.059621:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513442944:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781051101.060595:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051101.060602:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051101.060609:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443072:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:0.0:1781051101.060653:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443008:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051101.825332:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051101.953278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051102.849323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051102.977264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051103.873352:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051104.001264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051104.897347:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051105.025258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051105.921331:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051106.049277:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051106.177294:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051106.177303:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051106.177307:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051106.177320:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4742 00000100:00100000:1.0:1781051106.177321:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513443136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051106.177326:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051106.177334:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051106.177347:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051106.177364:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000006401720 x1867562513443264/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051106.177385:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051106.177389:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051106.177408:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051106.177418:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443264:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051106.178855:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443200:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051106.178867:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781051106.178873:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513443136:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051106.178874:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051106.178883:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443264:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051106.946247:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051107.073278:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051107.969305:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051108.097264:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051108.993325:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051109.121296:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051110.017293:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051110.145221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051111.041283:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051111.169229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051111.297322:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051111.297329:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051111.297333:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051111.297356:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4747 00000100:00080000:0.0:1781051111.297360:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051111.297366:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051111.297376:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781051111.297378:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513443328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051111.297396:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ce1ec364 x1867562513443456/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051111.297423:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051111.297427:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051111.297449:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051111.297539:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443456:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051111.298692:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009fa84b8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513443328:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051111.298708:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443392:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051111.298865:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051111.298870:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051111.298889:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ce1ec364 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443456:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051112.065295:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051112.193227:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051113.089346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051113.217253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051114.113340:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051114.241260:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051115.137324:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051115.265240:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051116.161346:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051116.289248:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:1.0:1781051116.417325:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781051116.417331:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1781051116.417334:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1781051116.417343:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4752 00000100:00080000:1.0:1781051116.417345:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1781051116.417351:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781051116.417352:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:1.0:1781051116.417362:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781051116.417396:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f1c3efb9 x1867562513443648/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1781051116.417418:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1781051116.417421:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781051116.417447:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513443520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051116.417458:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443648:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051116.418773:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000023b4b231 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513443584:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051116.418822:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1781051116.418823:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001d23a43b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513443520:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051116.418832:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051116.418842:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f1c3efb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443648:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051117.185328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051117.313268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051118.210415:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051118.337271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051119.233303:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051119.361267:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051120.257336:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051120.385253:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051121.281294:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051121.409263:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781051121.537350:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781051121.537357:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781051121.537361:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781051121.537373:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4757 00000100:00080000:3.0:1781051121.537377:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781051121.537385:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781051121.537394:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051121.537401:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513443712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051121.537413:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e4d6c6b1 x1867562513443840/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781051121.537433:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781051121.537437:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781051121.537478:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513443776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051121.537486:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443840:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781051121.538955:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513443712:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:2.0:1781051121.538981:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ed3dd285 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513443776:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051121.539034:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051121.539039:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051121.539044:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e4d6c6b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513443840:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051122.305294:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051122.433222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051123.329326:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051123.457247:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051124.353267:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051124.481201:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051125.377290:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051125.505224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051126.401284:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051126.529224:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781051126.657319:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781051126.657325:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781051126.657329:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781051126.657355:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4762 00000100:00080000:3.0:1781051126.657359:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1781051126.657366:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513443904:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:3.0:1781051126.657368:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781051126.657382:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:3.0:1781051126.657412:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000aa586f62 x1867562513444032/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781051126.657431:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781051126.657435:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:2.0:1781051126.657459:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1781051126.657468:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513443968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051126.657493:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444032:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781051126.658873:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051126.658880:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051126.658887:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000aa586f62 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444032:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781051126.658917:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513443968:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051126.658928:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513443904:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051127.425303:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051127.553216:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051128.449240:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051128.577238:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051129.473292:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051129.601221:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051130.497284:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051130.625234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051131.521280:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051131.649192:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:3.0:1781051131.777271:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781051131.777277:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1781051131.777279:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1781051131.777287:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4767 00000100:00080000:3.0:1781051131.777290:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1781051131.777295:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:3.0:1781051131.777303:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051131.777313:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513444096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051131.777317:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006e57f46b x1867562513444224/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1781051131.777331:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1781051131.777334:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781051131.777350:0:2216:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513444160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051131.777360:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444224:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:2.0:1781051131.778567:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d139f817 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513444096:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051131.778585:0:2216:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000029f3aabc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2216:1867562513444160:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:0.0:1781051131.778633:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051131.778638:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051131.778642:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006e57f46b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444224:192.168.201.144@tcp:8:kworker/u8:0.0 00010000:00010000:1.0:1781051132.545270:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051132.673232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051133.569302:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051133.697189:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051134.593301:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051134.721225:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051135.617299:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051135.745241:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051136.641297:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051136.769245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051136.897316:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051136.897322:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051136.897325:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1781051136.897336:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4772 00000100:00080000:2.0:1781051136.897339:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1781051136.897345:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513444288:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781051136.897347:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051136.897359:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051136.897373:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003fa10761 x1867562513444416/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051136.897401:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051136.897404:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781051136.897418:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1781051136.897424:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513444352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:0.0:1781051136.897450:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:0.0:1781051136.898702:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051136.898706:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051136.898710:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003fa10761 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444416:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781051136.898813:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000001d6480a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513444352:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:3.0:1781051136.898823:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c170795f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513444288:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051137.665327:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051137.793229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051138.689282:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051138.817244:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051139.713323:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051139.841284:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051140.737243:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051140.865246:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051141.761314:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051141.889208:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051142.017321:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051142.017328:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051142.017332:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051142.017344:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4777 00000100:00100000:1.0:1781051142.017344:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513444480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051142.017350:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051142.017359:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051142.017370:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051142.017388:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005b954604 x1867562513444608/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051142.017412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051142.017429:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:1.0:1781051142.017448:0:2214:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1781051142.017456:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513444544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051142.017464:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444608:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051142.018608:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000006401720 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2214:1867562513444544:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781051142.018619:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002db543f2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513444480:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051142.018688:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051142.018693:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051142.018697:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005b954604 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444608:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051142.785293:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051142.913273:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051143.809291:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051143.937194:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051144.833339:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051144.961237:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051145.857304:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051145.985255:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051146.881311:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051147.009242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051147.137302:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051147.137307:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051147.137309:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051147.137318:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4783 00000100:00080000:0.0:1781051147.137321:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051147.137326:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781051147.137331:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513444672:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1781051147.137335:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051147.137348:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000dd1a56a6 x1867562513444800/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051147.137365:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051147.137367:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051147.137385:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513444736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051147.137412:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444800:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051147.138636:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000728de7f4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513444672:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051147.138703:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051147.138720:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051147.138726:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513444736:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051147.138727:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000dd1a56a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444800:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051147.905274:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051148.033242:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051148.929311:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051149.057233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051149.953249:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051150.081249:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051150.977347:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051151.105222:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051152.001325:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051152.129252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051152.257296:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051152.257303:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051152.257307:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051152.257318:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4788 00000100:00080000:0.0:1781051152.257322:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051152.257328:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000100:00100000:1.0:1781051152.257331:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513444864:192.168.201.144@tcp:400:kworker/u8:3.0 00000008:00100000:0.0:1781051152.257340:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051152.257356:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000f82664a x1867562513444992/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051152.257404:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051152.257408:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051152.257427:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513444928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051152.257438:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444992:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051152.258797:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513444928:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781051152.258806:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000018617087 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513444864:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051152.258904:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051152.258909:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051152.258913:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000f82664a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513444992:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051153.025302:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051153.153232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051154.049343:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051154.177232:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051155.073322:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051155.201250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051156.097310:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051156.225245:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051157.121328:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051157.249233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051157.377435:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051157.377446:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051157.377450:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051157.377463:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4793 00000100:00100000:1.0:1781051157.377465:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051157.377468:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051157.377478:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051157.377492:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051157.377510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008a775f83 x1867562513445184/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051157.377544:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051157.377549:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781051157.377565:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445184:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051157.377573:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:1.0:1781051157.378887:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000059ddb857 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445056:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051157.378902:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445120:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051157.379310:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1781051157.379320:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:0.0:1781051157.379328:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445184:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051158.146702:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051158.273271:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051159.169336:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051159.297258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051160.193337:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051160.321233:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051161.217335:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051161.345585:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051162.241370:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051162.369268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051162.497340:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051162.497349:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051162.497353:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1781051162.497354:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:0.0:1781051162.497365:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4798 00000100:00080000:0.0:1781051162.497369:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051162.497376:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051162.497388:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:0.0:1781051162.497405:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000035676234 x1867562513445376/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051162.497431:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051162.497446:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051162.497467:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051162.497482:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445376:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051162.498623:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000398ab629 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445312:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051162.498757:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445248:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:1.0:1781051162.498798:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781051162.498802:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:1.0:1781051162.498805:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000035676234 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445376:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051163.265285:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051163.393268:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051164.289328:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051164.417258:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051165.313683:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051165.441257:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051166.337359:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051166.465212:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051167.361373:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051167.489270:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051167.617330:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051167.617341:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051167.617357:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051167.617368:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4803 00000100:00080000:0.0:1781051167.617371:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051167.617377:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051167.617388:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781051167.617389:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051167.617407:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000051f6c3cb x1867562513445568/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051167.617450:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051167.617453:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051167.617472:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051167.617562:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445568:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051167.619451:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008a775f83 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445440:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051167.619465:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445504:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:02020000:3.0:1781051167.619731:0:2213:0:(client.c:1361:ptlrpc_check_status()) 11-0: lustre-OST0000-osc-ffff9d7391e43000: operation ost_connect to node 192.168.201.144@tcp failed: rc = -19 00000100:00080000:3.0:1781051167.627214:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051167.627223:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051167.627234:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000051f6c3cb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445568:192.168.201.144@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1781051168.385285:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051168.513184:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051169.409357:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051169.537272:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051170.433339:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051170.561252:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051171.457330:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051171.585266:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051172.481362:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051172.609274:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:0.0:1781051172.737353:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051172.737359:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1781051172.737362:0:38:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1781051172.737371:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4808 00000100:00080000:0.0:1781051172.737374:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1781051172.737379:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:0.0:1781051172.737387:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:1.0:1781051172.737391:0:2214:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:0.0:1781051172.737399:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a1a7383 x1867562513445760/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1781051172.737415:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1781051172.737417:0:38:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:0.0:1781051172.737430:0:2215:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445696:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00100000:3.0:1781051172.737458:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445760:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1781051172.738587:0:2214:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004437141e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:a331c586-428d-4036-9702-69d922f5fec1:2214:1867562513445632:192.168.201.144@tcp:400:kworker/u8:3.0 00000100:00080000:3.0:1781051172.738623:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1781051172.738628:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051172.738632:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a1a7383 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445760:192.168.201.144@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1781051172.738675:0:2215:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003c180bf3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2215:1867562513445696:192.168.201.144@tcp:400:kworker/u8:3.0 00010000:00010000:1.0:1781051173.505280:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051173.633235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051174.529338:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051174.657219:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051175.553325:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051175.681250:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051176.577316:0:4993:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051176.705229:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00010000:00010000:1.0:1781051177.601266:0:4994:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051177.729235:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds 00000100:00080000:2.0:1781051177.857326:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1781051177.857333:0:8:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1781051177.857336:0:8:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1781051177.857340:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513445824:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00080000:2.0:1781051177.857346:0:8:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: connect to NID 192.168.201.144@tcp last attempt 4813 00000100:00080000:2.0:1781051177.857350:0:8:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1781051177.857358:0:8:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff9d7391e43000: import 000000006306ae4e using connection 192.168.201.144@tcp/192.168.201.144@tcp 00000008:00100000:2.0:1781051177.857368:0:8:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539520 ocd_grant: 66453504, lost: 0. 00000100:00100000:2.0:1781051177.857384:0:8:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000011c0ba5e x1867562513445952/t0(0) o8->lustre-OST0000-osc-ffff9d7391e43000@192.168.201.144@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1781051177.857420:0:8:0:(pinger.c:251:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: level IDLE/11 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1781051177.857423:0:2217:0:(ptlrpcd.c:415:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:2.0:1781051177.857424:0:8:0:(pinger.c:265:ptlrpc_pinger_process_import()) 7d9c7ecf-6041-42ed-aef7-34739e7e4960->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: IDLE) 00000100:00100000:3.0:1781051177.857432:0:2217:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513445888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781051177.857482:0:2213:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445952:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00080000:1.0:1781051177.858627:0:2213:0:(import.c:86:import_set_state_nolock()) 000000006306ae4e lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1781051177.858632:0:2213:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.144@tcp failed (-19) 00000100:00100000:3.0:1781051177.858634:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000521535b2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2217:1867562513445888:192.168.201.144@tcp:400:kworker/u8:0.0 00000100:00100000:1.0:1781051177.858638:0:2213:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000011c0ba5e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7d9c7ecf-6041-42ed-aef7-34739e7e4960:2213:1867562513445952:192.168.201.144@tcp:8:kworker/u8:0.0 00000100:00100000:3.0:1781051177.858651:0:2217:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000523c4981 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:a331c586-428d-4036-9702-69d922f5fec1:2217:1867562513445824:192.168.201.144@tcp:400:kworker/u8:0.0 00010000:00010000:1.0:1781051178.625298:0:10506:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1781051178.753234:0:19821:0:(client.c:2500:ptlrpc_set_wait()) set 00000000df080888 going to sleep for 0 seconds Debug log: 20444 lines, 20444 kept, 0 dropped, 0 bad.